Continue in 2 seconds

Now is the Right Time for Real-Time BI

  • September 01 2004, 1:00am EDT

Last September, when DM Review published my article entitled "BAM and Business Intelligence," the real-time business intelligence (BI) and business activity monitoring (BAM) marketplaces were very much in their infancy. Over the past year, real-time technologies have become more mature, our understanding of the use of real-time processing has improved and companies are beginning to implement real-time BI and BAM applications. My objective in this article is to review these changes and look at the benefits real-time BI processing offers.

Perhaps the biggest advance I have seen over the past year is that companies are beginning to understand that there are many different types of real-time processing, each with its own uses, benefits and issues. IT staff are also realizing that right time is a better term to use than real time. Right time implies that different business situations and events require different response or action times. When planning a right-time processing environment, it is important to match technology requirements to the actual action times required by the business - some situations require close to a real-time action, whereas with others, a delay of a few minutes or hours is acceptable.

There is a wide variety of definitions for the different types of right-time BI processing that exist, but most real-time BI requirements can be placed into one of four main categories: right-time data integration, right-time data reporting, right-time performance management and right-time automated actions.

Right-Time Data Integration

Right-time data integration supports the event-driven integration, propagation and migration of business transaction data and master data. One of the primary objectives of this style of real-time processing is to create a consistent and integrated store of business transaction and master data. The integrated store of data may be maintained synchronously or asynchronously with the source data, which in most organizations is dispersed across multiple systems. A synchronous data store will contain real-time current data, whereas an asynchronous store will contain low-latency data.

Right-Time Data Reporting

Right-time data reporting provides business users with on-demand access to business transaction and master data. The on-demand data may reside in current business transaction files and databases, or may have been extracted and integrated into a low-latency data store. The benefit of accessing current data is that it is real-time data. The advantage of accessing data that has been extracted into a low-latency store is that it is integrated and consistent. The downside of a low-latency store approach, of course, is that the data is not current - the data may be a few seconds or even several hours out of date, depending on the techniques used to maintain the store. For many applications, however, consistency is more important than timeliness.

Right-Time Performance Management

Right-time performance management enables business users to monitor and optimize the business performance of intraday business operations. It is one of three types of business performance management (BPM) - the other two types being strategic performance management and tactical performance management. The key attribute of right-time performance management (which is sometimes known as operational performance management) is that it is event-driven. As events occur in business transaction systems, they are sent to operational BPM applications for processing. These applications analyze the events and produce metrics about business performance in real time or at predefined intervals. Operational BPM applications can also be built using a data store created by right-time data integration.

Right-Time Automated Actions

Right-time automated actions improve the speed of business decision making and increase business productivity by automating the decision-making process. This is achieved by encapsulating business user expertise in a set of business rules that are embedded in a rules-driven workflow engine. As BI is produced (by a right-time performance management application, for example), it is passed to the rules engine for evaluation against associated business rules. These rules determine what action needs to be taken based on the results of the evaluation.

Technology Advances

Now that we have defined the four main categories of right-time BI processing, we are in a position to review the technology advances that have taken place over the past year in supporting them. The technologies we need to consider are BAM (business activity monitoring), BRE (business rules engines), EAI (enterprise application integration), EII (enterprise information integration) and ETL (extract, transform and load).

For right-time data integration, EAI and/or ETL can be used (see Figure 1). Although EAI technology was intended initially for application-to-application integration, it has also been used extensively for data integration and can, therefore, be used for building low-latency and consolidated master data stores. EAI technology is typically a component of an application integration platform, and leading vendors include BEA, IBM, Microsoft, Oracle and TIBCO.

Figure 1:  Right-Time Data Integration

The issue with some application integration platform products is scalability and transformation power, and this is why there are an increasing number of relationships between application integration and ETL vendors. The ETL products provide scalability and transformation power, while the application integration products provide an event-driven front end. Application integration products also offer a wide range of adapters for accessing back- and front-office application packages.

Over the past year, the ETL vendors have been enhancing their support for event-driven and Web services-based data sources. Leader charts from analyst companies such as Gartner, IDC and META Group show the key ETL vendors to be Ascential, Business Objects, DataMirror, IBM, Informatica, Microsoft, Oracle, Pervasive/Data Junction and SAS. The inclusion of Pervasive is important to note - its recently acquired Data Junction product set has enjoyed significant success in the small and medium business marketplace as a low-cost but powerful ETL capability. Many ETL vendors now call their products data integration platforms to demonstrate their move toward supplying a more generalized data integration environment for both batch and event-driven processing.

The key distinguishing factor between EAI/ETL product combinations will be the level of integration that is provided. Meta data management is likely to be a key issue here. It is interesting that some vendors such as IBM, Microsoft and Oracle provide both EAI and ETL solutions, and are improving the integration between the two types of products. Another direction to note is that application package vendors such as SAP are also integrating their EAI and ETL solutions. In the case of SAP, this integration is provided by NetWeaver, which is used by the SAP Master Data Management (MDM) product to provide a master data integration capability.

One technology that has gained significant attention of late is EII, which provides a federated query server that can retrieve and integrate data from multiple disparate data sources (see Figure 2). Some EII products also support access to unstructured data and message queuing software. The emphasis of EII is on optimizing access to heterogeneous data and on providing a single virtual view of that data. An EII server can be used to support the right-time data reporting of disparate business transaction and master data. An EII server can also be used to feed an ETL tool with data for building a data warehouse. Examples of key EII products include Actuate Enterprise Reporting Platform (incorporates EII capabilities acquired from Nimble Technology), BEA Liquid Data, Composite Software Information Server, IBM DB2 Information Integrator and MetaMatrix Server.

Figure 2:  Using Enterprise Information Integration (EII)

Given that EII is query-driven rather than event-driven, it is not ideally suited to the building of a low-latency data store. It is important to point out, however, that some EII product queries can be made event-driven. Data changes, for example, could be accumulated in a message queue and an EII query scheduled to run at periodic intervals to read the data from the queue and update a data store with the changes. I recently saw a demonstration where BEA Liquid Data EII queries were being executed from a BEA WebLogic workflow that was creating operational business performance management metrics. The queries were used to access planning information and historical business performance metrics maintained by Hyperion Software applications and databases. This architecture enabled the WebLogic application to display live performance metrics and, at the same time, show a comparison of this data against planning and historical information.

Operational Business Performance Management

Over the past year, there has been significant growth in the use of business performance management (BPM) applications and tools for deploying management dashboards that display strategic, tactical and operational performance metrics. For operational BPM, interest in the use and application of business activity monitoring (BAM) has increased, but the term is still misunderstood and poorly defined. Often, rather than use the term BAM, many industry pundits and vendors simply use the expression operational performance management or operational performance dashboards. These are more readily understood terms. In general, right-time performance management operational BPM and BAM can be considered to be the same thing.

The biggest development in BAM in the last year that has encouraged its incorporation into the overall area of operational BPM has been its focus, not just on its event-driven data integration infrastructure, but also on BAM applications that exploit that infrastructure. BAM can be broken down into two key components - a data integration infrastructure component that provides a scalable architecture for capturing and processing business events, and an application component that creates rules-based operational dashboards.

The direction of BPM is toward providing two types of dashboards: operational dashboards that display current business performance metrics and planning dashboards that show how these metrics align with tactical and strategic business goals (see Figure 3). Planning dashboards will be methodology-driven and/or will grow out of existing OLAP (online analytical processing) initiatives, whereas operational dashboards will be rules-driven. Operational dashboards provide an ideal mechanism for business users to understand and investigate the subtleties of day-to-day business operations. As experience with operational dashboards evolves, the business rules used to drive these applications will grow from simple rules based on individual metrics to complex rules that involve inter-metric business relationships. One key distinguishing factor between products here will be the power of their presentation capabilities. An excellent book on using dashboards to display business performance data is Show Me the Numbers: Designing Tables and Graphs to Enlighten by Stephen Few.

Figure 3: Right-Time Performance Management

Business Rules

As shown in Figure 4, there are several different types of business rules associated with BPM. Analysis rules are used to calculate performance metrics from detailed business transaction data, and context rules enable performance metrics to be tied to business goals and forecasts. Some basic BPM automation can be achieved by applying exception rules to metrics and sending an alert to a business user when a metric exceeds a threshold defined in the exception rule. Decision making can be further improved by including in the alert the Web address of a guided analysis workflow that identifies other analyses and reports that can be run to further investigate the issue.

Figure 4: Types of BI Business Rules

Full right-time automated actions can be achieved by defining the manual decision-making process that business users go through as a series of action rules in a workflow. These action workflows can then be implemented in a rules engine to automate the decision-making process. A rules engine may be embedded in a BPM application or may be an external business rules engine (BRE) product. The benefit of using an external product is that it can act as a central repository for defining and managing business rules. The issue with an external BRE is integrating it into the BI environment. One company that has been involved in both business intelligence and business rules engines is Fair Isaac. The company's decision management set of products is a good example of how BI and business rules technologies are beginning to converge.

The Right Time

The right-time BI marketplace is beginning to offer a wide range of different technologies and products. The key to success here is to build a smart right-time BI framework that incorporates these different technologies and to carefully evaluate right-time BI technologies to determine where in the organization more responsive business processes will achieve the biggest return on investment. Now is the right time to start evaluating the powerful benefits offered by right-time BI technology.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access