The BI-Search Evolution

Several years ago, I was involved in a consulting engagement with the IT operations of a multibillion dollar firm. The focus was on the areas of budgets, ROI achievement, effective system use and user satisfaction. While the IT staff did a good job of managing budgets, they often struggled to deliver strategic business benefits to the users. One of their biggest problems was the inability to deliver timely information to executives and business users who needed to make ad hoc decisions. Consequently, expensive MBA types scrambled to extract data from business intelligence reports and spreadsheets in order to prepare analyses for the managers.

A recent Forrester Research survey noted that when it came to BI environments, 59 percent of the respondents indicated that users were unable to access 100 percent of the data needed for reporting and analytic work. Additionally, 78 percent of the respondents indicated that their BI environment did not enable exploration and analysis with features such as adaptive data models, unlimited dimensionality and guided analysis.

Information access and exploration has become more challenging as more people attempt to make more decisions based on more data. The people who need answers can’t find what they need – nor can they easily use what they found. Companies have been implementing and using information systems for decades, so what makes this so difficult and expensive? The answer is apparent when you look at the underlying source systems and the process of bringing the data from those sources together:

Too many disparate data sources. New data sources are being introduced every day, and existing sources undergo constant changes. This complicates the task of unifying data and allowing for decision-making based on the most up-to-date information.
Evolving user needs. The information needs of users change as rapidly and continually as the business needs evolve. Users also have expanded data universes, moving beyond just spreadsheets to include all company-wide and even Web-based data.
Expensive and time-consuming data modeling. Often, users don’t know what information they need, so it’s difficult for them to articulate what data they require for decision-making. Because IT attempts to anticipate all the answers a user will ultimately want, this process often requires extensive data modeling in order to get the right answers.
Power tools for the everyday user. Many analytic tools are intended for sophisticated power users, but these users are only a small fraction of the decision-making population in a company. Today, almost every business user is expected to make informed decisions.

Fortunately, easy-to-use search and the power of BI are finally merging, so IT can now deliver on the promise of providing users with all the data necessary to make strategic business decisions, and the power to discover, explore and analyze.

Taken separately, traditional BI and enterprise search tools were each designed to solve problems other than ad hoc decision-making. BI was originally developed for reporting on structured data while search was designed for retrieving unstructured documents. As a result, each technology fall shorts in different ways in several key areas – a good user experience, the types of accessible information and the ability to respond to rapid change.

Because of its focus on reporting and structured data, BI tools are good at answering predictable questions and reporting on key performance indicators. However, they aren’t as effective at answering new or ad hoc questions, requiring the user to request custom reports and cubes from IT analysts.

This is because the rigid, hierarchical data models in BI tools only allow limited exploration and are often complicated to use. Even with rigorous data modeling, most BI tools cannot access unstructured content, and adding new data sources requires analysis and redesign of the data models as well as the reports, analytics and dashboards they drive.

On the other hand, everyone uses search today because, in many ways, it has become the simplest form of computing. Googling someone or using search on an e-commerce site or even on a company home page is the starting point of many users’ regular computer use. However, basic search, with its incomplete data model and document-centric retrieval, also allows for only limited exploration, depriving users of necessary context. Structured data is an afterthought. And, while new data sources can be easily added, providing context and exposing relationships to existing data is difficult.

However, BI and search can be combined to preserve the strengths of both and mitigate the drawbacks of each.

Enabling Discovery

To understand how combining search and BI can bring a richer solution to bear on business decisions, we have to first consider how humans actually make an ad hoc decision.

In daily decision-making, people formulate their next question based on the answer to a previous question. In the process, people often need help formulating good questions because they want to understand what the alternatives might be when making trade-offs. As they gain insight into their problem, they can use additional filters, graphs and visualizations to drill down and explore deeper.

In the business world today, people typically rely on BI systems to get an answer to a known business problem and rely on search to find information. Unfortunately, answers to either structured BI queries or text searches are only as good as the question posed. Although users might glean insight from the results returned in either case, they might never know if the question asked was the right one. Only a few business questions are simple enough to be served with a "hole-in-one" answer.

The convergence of BI and search technologies can enable a user to expose relationships in data that can often lead to an unanticipated answer or new revelation – without the necessity of the perfectly formed question.