Search has existed for a long time now, but with the success of Google, everyone is asking, why should accessing information be difficult anymore? Enterprises have long used business intelligence (BI) information successfully for decision support. Though search and BI appear to be distinct areas, they are complementary, and they can be fused for an effective pursuit of information within an enterprise. This presents a technical view of BI search solutions and discusses few scenario-based strategies for the implementation of a BI search solution.
Why BI Search?
The first question raised in building a business case for BI search is: why should an enterprise adopt BI search? Let us quickly look at some of the shortcomings of todays BI reporting. There is:
- No provision for natural language query;
- No support for relevancy based reporting; only binary notion supported; and
- No support for textually centric contents.
These deficiencies of BI reporting can be dealt with by BI search. In addition, BI search has its own set of advantages as revealed in an Internet-based survey.1 More than half of the total respondents chose the following as potential benefits:
- Self-service reporting and information discovery,
- Decisions based on broader range of information, and
- Greater user adoption of BI.
Besides providing a platform for self-service report discovery for consumers, BI search also encourages greater adoption of BI in the enterprise.
Yesterday, Today and Tomorrow
BI search has been in practice for many years now. In the past, BI systems offered search facilities only by the name of the report. It brought limitations to the BI systems by forcing end users to remember the report name. Imagine a BI system with several hundreds or thousands of reports. It was a daunting task to locate a BI report amongst such a mammoth set. In large enterprises with multiple BI platforms, searching for a BI artifact was next to impossible.
Today, the situation is totally different. BI search is driven by metadata. It provides search functionality over broad categories of report metadata, including report definition metadata and report structure metadata. Report definition metadata is a set of metadata that describes the identification of the report. It includes report name, author, etc. Report structure metadata describes the structure of the report. It includes report title, header, query, prompts, sections, tables, charts, alerts, filters, etc. With the empowerment metadata, search has been simplified today to the extent that to locate a report, one needs to know very little about that report.
Scope and coverage of BI search is evolving. The future roadmap of BI search promises features such as structured data exploration. This means users will have the capability of tracing a report to its source and examining the data at source instead of following the traditional way of running a report and fetching the data. Powered by such features, BI search has the potential to replace ad hoc query tools and help minimize the number of ad hoc reports created in the organization.
Scenario-Based Solution Strategies
BI search solution implementation provides quick access to all BI artifacts in the enterprise. The success of a BI search solution is heavily dependent on its implementation strategy. Before studying various implementation strategies, let us quickly examine two scenarios that can exist in an enterprise:
- The enterprise has standard BI platforms or
- The enterprise has homegrown BI applications.
There is no single correct strategy for the implementation of a BI search solution. To ensure a smooth and successful implementation, the enterprise BI landscape must be appraised; and then the strategy and implementation option that best suits the given scenario must be selected. For each of these two scenarios, recommended strategies and implementation options are discussed in detail in the following sections.
Standard BI Platforms
Many enterprises, large businesses in particular, have standardized their BI reports generation using one or more industry-standard BI tools. Conventional BI tools offered simple search against report name. They lacked metadata-driven comprehensive search. In light of search-enabled BI, vendors are now offering a complete search service that is metadata centric. However, the limitation here is that BI vendors search offerings only work within their platform. Large enterprises are likely to have two or more standard BI platforms. In order to cover up compatibility issues, BI vendors are partnering with search vendors.
When an enterprise has two or more standard BI platforms and each one of them provides a search service in silos, it is recommended to perform the search in a distributed manner and collate the search result as depicted in Figure 1. In this solution strategy, either search is performed programmatically in each BI platform using its search services, or an enterprise search engine leverages the BI platforms search services to perform the search, and the results are combined and presented to the end user through a single user interface.
When implementing a BI search solution for standard BI platforms, two primary options exist.
The first option is to custom build a wrapper that acts as an interface between the end user and enterprise BI search engines/services. A wrapper is a custom piece of code/program. That provides coverage across all BI platforms. It receives the search query from the user interface, routes it to enterprise BI platforms, collects the query results from each BI platform, collates the results and sends them back to the user interface. This wrapper is generally developed using service-oriented architecture or application programming interface/ software development kit-based programming.
The second option is to let the enterprise search engine leverage all the BI search engines/services. As a result of alliances between enterprise search vendors and BI vendors, enterprise search tools support connectivity to BI platforms as well. However, the caveat here is that a search vendor product may not be compatible with one or more enterprise BI platform(s). Hence, it is always advisable to evaluate all the prominent search vendor products and shortlist few. Doing a proof-of-concept prior to placing the purchase order will help ensure the suitability of the product to the enterprise environment.
Homegrown BI Applications
Historically, BI applications were developed using conventional application methodology with embedded analytical capabilities. This offered a greater degree of control to the enterprise IT department; however, it failed to offer flexibility to end users, and they were forced to rely on IT personnel for their ad hoc requirements. This posed difficulties to the enterprise IT department from a maintenance perspective. Though off-the-shelf BI products ought to replace in-house or homegrown BI applications, homegrown applications continue to exist today.
To provide a BI search capability in an enterprise environment where the business is relying on a number of homegrown BI applications for analytics, it is recommended to integrate report metadata across these applications. In this solution, metadata from all BI applications is extracted, cleansed and populated in a centralized database called a search repository in a searchable format - upon which the search operations are performed. An architectural anatomy of this solution is presented in Figure 2.
In this solution strategy, the enterprise has a single search engine that crawls across all BI applications, extracts report metadata and populates it into a centralized database. This metadata is then indexed by the search engine to expedite the search process. Various components involved in this architecture and their functionalities are briefly described below.
- Content repositories - BI data stores that supply report metadata.
- Spider - It extracts report metadata from all BI content repositories.
- Indexer - It indexes report metadata and stores it in a search repository.
- Search repository - A data store for indexed report metadata.
- Content analyzer - It infers a query using techniques like lemmatization, spell check, etc. and performs search on a narrow set of contents.
- Content filterer - It locates matching BI artifacts for the user query.
- Search engine - A logical collection of various components.
- Query - A set of keywords that defines the user requirements.
- Result A collection of links that point to various BI artifacts.
As depicted in the concept diagram Figure 2, only report metadata is extracted from BI content repositories. Even though it is technically possible to provide search over report data values, it is generally not preferred due to its highly volatile nature.
When an enterprise has homegrown BI application, the first thing that needs to be ensured is the availability of comprehensive report metadata. Yesteryears applications may not have been designed to capture report metadata. In such cases, application(s) should be revamped to accommodate report metadata. Once the metadata is in place the simplest option for implementation would be to embrace an enterprise search tool.
Search vendors provide native database connectivity to a range of databases. Enterprise search tools can hook into a homegrown BI applications repository, extract the report metadata and index it for faster retrieval. The positive side of tool-based implementation is that development time of the solution will be very minimal. However, on the flip side, product cost could be unreasonable for a core BI search solution implementation. Besides enterprise search engine implementation, one can consider a set of options that are listed in Figure 3.
Custom-built search engine and metadata repository. The first option is about creating an integrated, centralized metadata repository and building a powerful search engine from scratch. It provides industries specific custom features and greater flexibility; however, it is a complex and time-consuming task. Any slippages in timelines will directly impact development cost.
Custom-built search engine over vendor metadata solution. The second option is to build a custom search engine over the enterprise metadata repository that is implemented using metadata vendor tool e.g., ASG Rochade or Informatica Metadata Manager (SuperGlue). The advantage of this option is that metadata integration at the enterprise level can be expedited using vendor tool. It is scalable and can easily accommodate other BI metadata like extract, transform and load metadata for example. However, interoperability of a custom built search engine with a metadata vendor tool may remain a challenge.
Vendor search engine on top of custom metadata repository. The third option is similar to the first one in the sense that an integrated, centralized custom metadata repository is built, but it is different in the sense that an enterprise search vendor like Autonomy, Endeca or FAST search engine is deployed on top of it. On the positive side of this option, development time is reduced dramatically, the customized set of BI report metadata is captured and the metadata repository can be easily integrated with the vendor search engine to index BI report metadata. On the negative side, maintaining security across BI applications could be a challenge.
Vendor search engine on top of vendor metadata solution. The fourth option is similar to the second one in the sense that the metadata vendor tool is used to integrate metadata from all BI applications, but it is different in the sense an enterprise search vendor product is used to index metadata instead of a custom-built search engine. Advantages of this option include scalability and performance. However, the extremely expensive nature of this option and product interoperability could be considered drawbacks.
Conventional BI systems offered limited or no search capability, but this situation is changing. Fusion of BI and search is gaining traction because it offers metadata-driven search capability in and across all BI platforms. To ensure success of a BI search implementation, the enterprise BI environment must be assessed and the strategy and implementation option that best suit the enterprise situation must be employed.
- Phillip Russom. BI search and Text Analytics: New Additions to BI Technology Stack. TDWI Research, April 11, 2007.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access