The Next Generation of Business Intelligence: Operational BI
A business intelligence (BI) system is a key component of a company's IT framework. It is the component that enables business users to report on, analyze and optimize business operations to reduce costs and increase revenues. Most companies use this component for strategic and tactical decision making where the decision-making cycle may span a time period of several weeks (e.g., campaign management) or months (e.g., improving customer satisfaction).
Competitive pressures, however, are forcing companies to react faster to changing business conditions and customer requirements. As a result, there is now a need to use BI to help drive and optimize business operations on a daily basis, and, in some cases, even for intraday decision making. This type of BI is usually called operational business intelligence.
The objective of this article is to explore the business requirements for operational BI and to review different types of operational BI processing, technologies and products that support those requirements. The objective of operational BI is to make more timely business decisions, and, therefore, it has a close relationship to the subject of real-time or right-time BI processing.1
Types of IT and BI Processing
IT systems support three main types of application processing (see Figure 1): business transaction (BTx) processing, BI processing and collaborative processing. BTx processing drives day-to-day business operations and supports business activities such as order entry, inventory control, shipping, billing and so forth. BI processing reports on and analyzes BTx processing, and provides information about how well this processing is meeting business requirements. Business users employ the output produced by BI applications to optimize BTx processing to more closely match business goals and requirements. This optimization process involves discussions between business experts about possible ways of improving business processes. The interaction between these business users is enabled by collaborative processing.
Figure 1: Types of Business Processes and Data
In a traditional BI system environment, the time between events occurring in BTx systems and action being taken based on BI system output is relatively long - a matter of days, weeks or months. The strategic and tactical decision making supported by a traditional BI environment is reactive in nature and is based on summarized and historical data. This long decision-making cycle allows the BI system to be loosely connected to related BTx and collaborative applications. Batch ETL jobs can be used to extract operational source data and load it into a data warehouse; and reporting applications can be used to produce and burst reports, and distribute the results to Web-based desktop and mobile users.
The objective of an operational BI system is to react faster to business needs and to anticipate business problems in advance before they become major issues. This style of processing requires tighter connections between the BI system and its BTx and collaborative counterparts. It also requires more timely (i.e., zero- or low-latency) detailed BTx data. How timely this data needs to be will vary by company and application. Telephone and credit card companies using BI to detect fraud will need the data to be as close to real-time as possible, whereas the use of BI to optimize supply chain and procurement operations is less time-sensitive.
Operational BI Reporting
Operational BI is not new -- companies have been doing operational reporting for many years. When mainframes dominated the IT landscape, operational reporting consisted primarily of batch production reporting jobs that were run overnight against live BTx data to avoid affecting the performance of online BTx processing. Prime-time ad hoc reporting was usually restricted to parameterized queries that were used to retrieve information about a specific customer, order, product, etc. In this environment, production and ad hoc report output often contained encoded information, such as product codes, that only a business expert could interpret.
As the number of data sources proliferated and business needs caused online operations to be extended (approaching 24 hours in some cases), it became clear that another solution had to be found for supporting operational reporting. One popular approach is to collect and consolidate detailed BTx data into an operational data store (ODS). The use of an ODS allowed batch and ad hoc operational reporting at any time without directly affecting online BTx application performance. It also provided a single and integrated view of BTx data, and enabled the BTx data to be transformed into a more usable and readable format.
The downside of using an ODS is the performance impact on BTx processing when capturing data changes for consolidation into the ODS, and the need to store and manage duplicate data. Another disadvantage for certain types of applications is the data latency introduced by copying data from BTx data sources into the ODS. It is both difficult and expensive to create a close to real-time ODS. Data warehouse appliances from companies such as Datallegro and Netezza can help reduce the cost of managing ODS data where the amount of data is in the order of one to three terabytes. For larger ODS environments, solutions from companies such as Teradata are more appropriate.
Another solution for doing operational reporting is to replicate the BTx source data into a second identical copy. The replicated copy is usually a real-time copy that can be used for disaster recovery, in addition to operational reporting. The problem here, of course, is that this approach doesn't support the reformatting of BTx source data and doesn't overcome the issue of having to report against multiple dispersed BTx data stores.
The advent of BTx application packages from companies such as Oracle and SAP helped solve some of the problems associated with operational reporting. These companies offer suites of BTx applications that are based on a single logical business view of BTx data. This approach provides a simpler view of BTx data and, in theory, reduces data proliferation. These application packages also often contain operational reporting applications that reduce the need to build custom applications. Even though reporting against live operational data still impacts BTx application performance, improved price/performance of computer hardware does allow a company to reduce this performance impact by using a larger hardware configuration.
Although many companies have moved toward the use of application packages, it will be years before many of them completely eliminate their legacy applications. In theory, a company would normally employ a single suite of BTx applications and a single logical BTx data store, but this is usually not the case. Frequently, different parts of a company may use different application package modules. Also, because of mergers, acquisitions, etc., a particular module may used and replicated in several different parts of the organization. Use of application packages often involves significant customization, which makes it difficult to migrate to new releases of a particular application package module. The net result is that many companies not only have legacy applications and application packages, but they also have multiple copies of the same package at different release levels. This proliferation of BTx data makes it difficult to do operational reporting.
An ODS is again a possible solution to the data proliferation problem, and some application package companies offer ODS support in their business intelligence solutions. One of the best examples of this is SAP Business Intelligence.
A recent addition to BI technology is enterprise information integration (EII), which provides a virtual business view of dispersed BTx data. To applications and business users, the data appears to be in a single data store, but in reality, it is still in its original source location. The role of the EII server is to access the various underlying BTx data sources to satisfy application and user queries issued against the virtual business view. This approach is often called a federated data architecture (see Figure 2). EII works well when the federated queries are very specific in nature, when the amount of data returned is small and when the source data does not require significant reformatting and cleaning.
Figure 2: Types of Data Integration
In summary then, operational data reporting can be done using live, consolidated, replicated or federated data. The approach used will depend on both business and technology requirements. A company's data architecture should, however, be capable of supporting all four approaches.
Operational BI Analysis and Performance Management
Operational reporting applications simply format and display the content of BTx data stores. Many business users, however, want to be able to analyze BTx data to identify potential problems and look for business trends. There are a wide variety of analysis tools available on the market. The capabilities provided by these products range from using Microsoft Excel for data analysis to advanced OLAP applications and database systems.
The biggest growth area in operational BI analysis is in the area of business performance management (BPM). Operational BPM applications not only analyze the performance of BTx processing, but also compare the measured performance against business goals and alert business users when actual performance is out of line with business goals. The context for the comparison (i.e., the business goals and objectives) may come from planning (e.g., budgeting, forecasting) systems, a traditional data warehouse or user-defined rules. The output from BPM applications may be presented to the business user in an operational dashboard, portal, e-mail message, etc.
The source data to be analyzed is the same as that used for operational reporting, and so the live, consolidated, replicated and federated data approaches mentioned earlier apply equally to the BI analysis and BPM environment. Some BPM tools have the capability to monitor and analyze business events flowing through a BTx application system. Application integration vendors such as IBM and TIBCO provide such a capability, as do independent third-party BPM vendors such as Celequest. In some products, the events are cached in memory and do not have to be made persistent in a data store (such as an ODS) before they can be analyzed. This approach extends data consolidation to what can be thought of as an event-driven data architecture (see Figure 2). This architecture is particularly useful for applications that require close to zero data latency.
A Process-Driven Approach
Most designers and developers of BI and data warehousing applications come from a data centric background. In the operational BI environment, however, the objective is to report on and analyze business processes and their underlying activities. It is very important, therefore, that BI reports, analyses and dashboards be process centric, not data centric.
Operational BI is not only intended for business analysts, but also for executives, managers and line-of-business users. Sales managers and customer support staff need information to be presented in a form that relates to the business tasks and activities they perform in their everyday jobs. They not only need information, but also workflows and guided analyses that help them interpret and analyze this information. This requires a process-centric, rather than data-centric approach to BI processing.
Operational BI must be tightly connected to business processes. BTx application developers and vendors understand this need, but many BI developers and vendors do not. There is, however, growing interest in the use of business process management in BI. It is interesting that business performance management and business process management share the same BPM acronym. As we move forward, these two technologies will become closely intertwined and perhaps a better acronym would be BPPM (business process and performance management).
Political and Organizational Issues
The need to tightly integrate BTx, BI and collaborative processing raises some important organizational and political issues. Operational BI applications can involve both the BTx development group, and the BI and data warehousing development group. This can lead to demarcation disputes about who owns the project (and the budget!) and about the technologies and products that should be used. To overcome these problems, some companies have started to eliminate the distinction between BTx development and BI development.
Strategic and tactical BI is well entrenched in most organizations, and the biggest growth area in BI over the next few years will be in operational BI. This type of BI promises significant business benefits and extends the use of BI to a much wider user audience.
- Colin White. "Now is the Right-Time for Real-Time BI." DM Review, September 2004.