In the current difficult economic climate, corporations are looking to business intelligence (BI) to help them reduce costs and improve business efficiency. One new BI technology that promises to be of benefit is business activity monitoring (BAM), a term used to describe a BI approach that "provides real-time access to critical business performance indicators to improve the speed and effectiveness of business operations."1

BAM extends business intelligence system usage beyond strategic and tactical business decision making to the management of day-to-day business operations. Gartner, Inc. predicts that "by 2004, in enterprises where faster reaction is key to operational effectiveness, BAM will be one of the top four initiatives driving IT initiatives and strategy." There is, however, a lot of confusion about exactly what BAM is and how it relates to current BI approaches and methodologies. In this article, I'll try to clear up the confusion by providing an overview of BAM and its business benefits and describing how BAM technologies and products are integrated into an existing BI framework.

The BI Story So Far

To provide some perspective, let's briefly look at how BI techniques have evolved to date before looking at BAM in detail. I'll use the business intelligence framework shown in Figure 1 to guide the discussion.


Figure 1. Business Intelligence Framework

Initially, BI systems consisted of the extract, transform and load (ETL) server and reporting tools shown in Figure 1. The ETL server was used to extract data from operational systems and load it into a data warehouse, and reporting tools were used to give business users the ability to run canned, parameterized and ad hoc reports. Data warehouse developers and users quickly realized that they needed more than just basic reporting tools if they were to obtain good return on investment (ROI). This requirement led quickly to the development of analysis solutions such as OLAP tools and engines, and multidimensional data stores. More recently, these BI analysis tools have evolved to provide Web-based information delivery, and the use of business intelligence portals and executive dashboards provides easy and personalized access to business analytics and reports.

Another recent trend in BI analysis tools has been to add a performance management capability that enables business users to compare the analytics produced during BI processing to actual business goals and forecasts (i.e., it puts BI into a business context). Performance management products extend the use of BI from measuring business performance to managing it. The actionable intelligence produced by these products is presented in the form of drillable scorecards that employ formal or informal methodologies to document business goals and initiatives. Some performance management products also provide rules-driven facilities to send alerts to business users when thresholds defined by the user are, for example, not achieved. Alerts reduce the amount of time business users spend in accessing and analyzing data, and reduce the reaction time required to identify and fix business problems.

The tools discussed thus far enable business users to react to business situations after they occur. The predictive tools component shown in Figure 1 adds techniques such as data mining and forecasting to a business intelligence framework. These techniques help users to become more proactive in managing the business. In some cases, predictive tools are used to provide the business context (e.g., rules, forecasts) for the scorecards and alerts used by performance management tools.

Operational Business Intelligence

The BI framework shown in Figure 1 is used primarily to manage strategic and tactical business performance. Many organizations, however, would also like to use their BI systems to monitor and manage day-to-day business operations and performance in order to be able to react more rapidly as business circumstances change. This type of business intelligence processing requires additional capabilities to those described earlier for strategic and tactical business performance management.

As a first step in supporting operational business intelligence, vendors are beginning to provide facilities to create a low- latency store of integrated operational data. This so-called operational data store (ODS) has a data latency (compared to the data in operational systems) that is usually less than one day, or maybe even a few minutes or seconds. An example of ODS usage is a single customer store that reflects and integrates all customer interactions across a company. The actual data latency of such a store is usually a trade-off between business requirements and the cost of supporting low-latency data because implementation and operational costs increase significantly as data latency needs approach real time. An ODS can also act as a data source for predictive tools and, in some cases, the population of a data warehouse.

The growing need to integrate operational data into an ODS has led to ETL vendors extending their products into a data integration platform that replaces the batch loading process of data warehousing by a server that streams data from operational systems into an ODS. These platforms support additional data sources such as Web clickstreams and logs, as well as application integration software (such as messaging middleware) that is ideally suited for tracking and routing event- based data.

Business Activity Monitoring

The facility that is currently missing from operational business intelligence solutions is performance management. Existing techniques for data analysis and performance management are not ideally suited to managing business operations at an operational level, especially when performance management needs to be done in close to real time. BAM is designed to fill this gap.

A framework for BAM processing is shown in Figure 2. At first sight, this framework appears to be similar to that shown in Figure 1, but there are some important differences. The first difference is that the ETL server is replaced by a BAM server, which monitors and captures events that modify the state of operational processes. These events can be recorded in a real-time store (RTS), which can be used as a data source for building a data warehouse.


Figure 2. Business Activity Monitoring (BAM) Architecture

Events captured by the BAM server may be data events or messages such as those captured by an ETL or data integration platform server, but they may also be events occurring in hardware devices such as an ATM or POS terminal, or application events such as a business transaction, call to another application component, and so forth. An ODS can also act as a source for a BAM server.

BAM applications monitor day-to-day business processes such as customer orders, insurance claims and supply chain operations. BAM products are typically driven by process models. This is very different from data-driven ETL applications, which have little or no knowledge of business processes. (Aside: it is worth mentioning that strategic and tactical performance management is also likely to become business process driven in the future.)

As a BAM server tracks operational events, it maintains these events in a cache that is used by a reporting and analysis engine running under the control of the BAM server. The analysis engine can access existingbusiness intelligenceand datawarehouse information to put theoperational events being tracked intoabusiness context and produces scorecards of operational business performance. Some BAM environments, however, also provide the ability to do more detailed analysis and mining of information in the RTS.

Performance reports and scorecards produced by the reporting and analysis engines can be sent to a display console, dashboard or a portal. They can also be processed by an associated rules engine that can send appropriate alerts to business users, or action messages and transactions to operational systems. All BAM processing and analysis is done in a single integrated environment that provides scalability, reliability and high performance.

The key benefit of a BAM environment is that operational processes can be monitored and exceptions acted on in close to real time. Examples of business applications that lend themselves to BAM are summarized in Figure 3.

Figure 3. Examples of BAM Applications

Finance

Customer portfolio management
Programmatic trading
Fraud detection
Risk management
Real-time marketing and promotions
Automated loan and credit card application processing
Patriot Act compliance

Manufacturing

Order management
Product recalls
Quality control and monitoring
Sales forecasting and pipeline monitoring
Just-in-time inventory monitoring and parts receivable

Retail

Single customer store
Real-time inventory analysis
Real-time marketing and promotions
Product recalls

Healthcare

Real-time notification of patient status
Medical alerting about area emergencies, disease outbreaks
Claim processing and fraud detection

Transportation

Ticket pricing
Aircraft performance monitoring
Fraud detection
Freight car tracking and routing

The BAM Product Marketplace

Many different software companies are targeting the BAM space, including business intelligence vendors, DBMS vendors, application integration vendors and independent BAM vendors.

As we have already discussed, business intelligence vendors are modifying their products to support the population of a near real-time store such as an ODS. Reporting tools can be used with an ODS to provide a basic BAM capability, but data integration platforms need to be extended to understand business processes and application events in order to support a full BAM environment. Examples of BI vendors that are focusing on BAM include Ascential Software with DataStage and its new Real Time Integration (RTI) Services, and Informatica with PowerCenter RT and its new Business Activity Platform (which is a joint development with WebMethods). The challenge for BI vendors will be to support a scalable environment that can support a process-driven approach to performance management.

DBMS vendors often market their products for building a large-scale and near real-time ODS. These vendors, therefore, are candidates for supporting BAM.Two vendors that are particularly focused on BAM are HP with its ZLE solution and NonStop SQL/MX, and Teradata with its Active Data Warehouse and Teradata Database. These two products are usually considered to be at the extremes of the application spectrum ­ HP at the high-volume transaction end, and Teradata at the large data warehouse and complex query end. Both vendors, however, now market their products for supporting the mixed workloads of a BAM environment and have made product enhancements to support such an environment.

BAM workloads add significant short transaction processing requirements to a database system; and even though support for mixed workloads is essential for success in BAM, it is often difficult for a database product to be able to concurrently handle both high-volume BAM and high-volume complex data warehousing workloads. The decision whether to use an OLTP-centric or a complex-query-centric database engine for BAM will depend on the rate of change within the underlying data, the degree to which these changes need to made immediately visible to users and the users' need for summarized versus detailed data.

Application integration vendors such as BEA, IBM, Microsoft, SeeBeyond, Sybase,Tibco, Vitria and WebMethods currentlyoffer software that is used to track and route application and data events. Many of these vendors are adding, or acquiring, process management software to support business process management. Given their support for process-driven event management, it is a logical direction for application integration vendors to support BAM. The challenge for these vendors will be to integrate their BAM capabilities into existing business intelligence environments. This is why we are seeing relationships being built between application integration vendors and BI vendors. The benefit for application vendors is BI integration, and the benefit for BI vendors is support for application events and process management.

The BAM market is also seeing the emergence of new vendors such as Celequest (a company founded by Diaz Nesamoney, one of the original founders of Informatica) and Praja (recently acquired by Tibco) who are building BAM products from the ground up. The advantage offered by these vendors is that their products are designed to provide a single integrated environment for BAM processing. This approach has obvious advantages over other solutions that have to be retrofitted into a BAM environment. The independent BAM marketplace will grow; and, as with all new marketplaces, many vendors will go out of business, others will be acquired and the market will be eventually dominated by a few key players.

Preparing for BAM

Given that BAM is likely to become a key component in your organization's future business intelligence framework, how do you prepare for it?

First, it is important to gain a good understanding of BAM technology and its business benefits. Whereas BAM can provide significant business ROI, it is important to realize that it can be expensive to implement, particularly in a high-volume and low-latency environment. Of course, not all BAM applications require near real-time processing. A latency of several minutes, or even several hours, may be quite adequate for many BAM implementations. Regardless, a clear business case and ROI must be established for a BAM project. It will, therefore, be important for IT managers to discuss the benefits of BAM with line-of-business managers to determine where BAM would bring the most business benefit (for example, customer relationship management, procurement, manufacturing and distribution).

Second, BAM must be integrated with existing business intelligence and enterprise integration solutions. Gartner often describes BAM as the convergence of operational BI and real-time application integration. Without this integration, yet another information silo will be created. It will be essential to understand the facilities being offered by vendor BAM products and what interfaces are provided for enterprise and BI integration. Reporting on a near real-time ODS may be sufficient for basic BAM applications, for example.

Third, it is important to realize that although there are already a number of successful BAM implementations, this is leading-edge and immature technology that will undergo rapid evolution over the next few years. It will be important to track this evolution and update BAM strategies and BI frameworks accordingly.

Last, it is crucial to recognize that BAM is not just about technology. For BAM to be successful, organizations must modify their business practices and educate business users about BAM in order to exploit and to gain maximum business benefit from BAM initiatives.

1 David McCoy. "Business Activity Monitoring:
Calm before the Storm." Gartner document LE-15-9727, April 2002. This document can be found at: www3.gartner.com/resources/105500/105562/105562.pdf.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access