We live in real time, minute by minute. News is no longer delayed by days or even hours; it is streamed in real time. We bank online and check our real-time balances. We book flights with real-time visibility of seat availability. Sales patterns change over time and from place to place. Currency valuations shift and alter profit margins. Balancing on this shifting terrain, business managers are expected to focus on business analytics and make informed business decisions. Real-time business intelligence ensures accurate data flows across the enterprise so that organizations can make quick decisions on pricing, shelving, service and product mix, based on the latest information.

According to Dr. Richard Hackathorn, creator of the Time-Value Curve, “the value of data is directly proportionate to how fast a business can react to it. In other words, a corporation loses money every time it delays getting information into the hands of decision-makers.”

Real-time BI is crucial to survive in this competitive world. It is important to understand the new challenges that must be addressed and develop a solution that will handle the requirements and technology hurdles at hand.

Real-Time Business Intelligence

The major goal of real-time BI is reducing the time taken for corrective action or initiative. Real-time BI is designed to control data latency, analysis latency and action latency. Companies must understand that ROI will also depend heavily on the ability of an organization to modify its business practices to take advantage of improved responsiveness in the IT system.

A real-time BI system has two main components: real-time data integration and real-time decision-making. The objective of the real-time data integration component is to capture business events from operational systems and integrate them into a low-latency store. This component supports real-time, data-on-demand processing. The real-time decision-making component, on the other hand, supports real-time performance management and real-time predictive analysis. Figure 1 gives an overview of real-time BI architecture.

Challenges of Real-Time BI

BI applications include the activities of decision support, query and reporting, online analytical processing, statistical analysis, forecasting and data mining. Each of these components needs to be designed to operate in a real-time environment, and there can be many challenges in designing such system. Some major challenges include:

Designing real-time ETL. Traditional ETL tools are batch oriented, wherein the data becomes available as some sort of extract file on a certain schedule, usually nightly, weekly or monthly. Then the system transforms and cleanses the data and loads it into the data warehouse. ETL tools tend to update systems with complete files, not compact amounts of change data. However, for real-time ETL, a continuous flow will be required throughout the day with minimum latency. Real-time operation requires the synchronization of data across multiple layers of an organization and many different sources. Connecting the large and diverse array of data sources to a real-time warehouse is highly complex.

Data modeling for real time. From a data architecture perspective, real-time data warehousing challenges the posture of the data warehouse as a system of periodic measurements, advocating the requirement for a system of more comprehensive and continuous temporal information, i.e., a real-time database model that deals with the temporal nature of data.

Search, OLAP, and query and reporting. Today's query and OLAP tools, not having been designed with real-time warehousing in mind, can produce unanticipated results.

Scalability. To support real-time processing, the system must have a scalable and flexible back-end database environment for loading and administering large amounts of data. The database must also be able to handle mixed workloads, since the tasks used to update low-latency stores will need to run in parallel with real-time decision-making applications. Real-time processing will involve real-time alert reporting through emails or messages. These alerts need to be designed to operate on real-time data feeds.

Suggested Solutions

An ideal real-time BI tool will be the one which can answer all the above challenges. Experts all around the world are studiously working to develop such a system and have come up with many approaches to design a real-time BI. Some approaches are briefed here.

Micro batch ETL. A data warehouse can only be considered real-time, or near real-time, when all or part of the data is updated, loaded or refreshed on an intra-day basis, without interrupting user access to the system. Convention ETL, file based approach is extremely effective in addressing daily, weekly and monthly batch reporting requirements. Micro batch ETL designed on log based, real-time change data capture technology can provide a nonintrusive means for real-time data acquisition from an operational data source. Figures 2 and 3 give a pictorial presentation of the system.

Log-based change data capture technology captures data changes in the source system as they happen and flows them immediately to the target system,, ensuring business information is always reliable and timely. Most database management systems manage a transaction log that records changes made to the database contents and to metadata. By scanning and interpreting the contents of the database transaction log, one can capture the changes made to the database in a nonintrusive manner. The principal task of the CDC process is to scan the log and write column data and transaction-related information to the CDC change tables. It detects when tables are newly enabled for CDC and automatically includes them in the set of tables that are actively monitored for change entries in the log. Similarly, disabling CDC will also be detected, causing the source table to be removed from the set of tables actively monitored for change data. When processing for a section of the log is finished, the capture process signals the server log truncation logic, which uses this information to identify log entries eligible for truncation.

 

Enterprise Application Integration

EAI is the use of software and architectural principles to integrate a set of enterprise computer applications. Supply chain management, customer relationship management, BI and other types of applications typically cannot communicate with one another to share data or business rules. EAI is the process of linking applications within an organization together in order to simplify and automate business processes to the greatest extent possible, while at the same time avoiding sweeping changes to the existing applications or data structures. Functioning as a central hub, the EAI server transforms data from one application to another and can coordinate processes among applications. This means, for example, that when a bank customer changes the name on a personal checking account, it automatically changes on all of the customer's accounts. The EAI system may also send a message for an insurance representative to contact the customer, because name changes often signify a change in marital status. Figure 4 represents the EAI system pictorially.

EAI provides a vehicle for pushing data from source systems into the data warehouse. EAI can be used to facilitate data acquisition directly into a near real-time data warehouse or to deliver decisions to the online transaction processing systems that will be responsible for the associated bookkeeping activities. From a data integration perspective, EAI can be used to transport data between applications and to route real-time event data to other data integration applications like an ETL process. Access to application sources and targets is done via Web services, Microsoft .NET interfaces, Java-related capabilities such as JMS, legacy application interfaces and adapters, among others. There are two fundamental components:

  • Adapters map heterogeneous data formats, interfaces and protocols into a common model and format. The purpose of adapters is to hide heterogeneity and present a uniform view of an underlying heterogeneous world. A different adapter is needed to integrate each type of application.
  • A message broker facilitates interaction among adapters and, therefore, among the back-end systems that need to be integrated.

Enterprise Information Integration

In most enterprises, information is stored in separate databases, data warehouses and applications. EII products make it possible to combine information from these different data sources on demand. EII creates a layer of abstraction between the applications requiring the information and the source systems where details about data structure differences, data location, data sources and security differences are hidden. EII acts as a pull engine, which waits for the requests, splits the query across multiple heterogeneous data source systems, gathers transactional data sets, merges them together and then pushes them out to the requesting applications. These requesting applications can be a Web service, Excel or some other front end. EII makes information available in real time to a variety of applications and greatly simplifies and speeds access. Different views can be created for different applications or types of users. EII creates a layer of abstraction between the applications requiring the information and the source systems where details about data structure differences, data location, data sources and security differences are hidden. EII works well when a limited amount of data is required to support query processing, such as operational reporting or discrete data integration. EII technology is often used in retail to support call center operations. When a customer contacts customer service, the customer service desktop uses EII software to query multiple source systems for the specific details for that customer. Figure 5 shows the overview of the EII system.

In the future, real-time BI will experience a change with broad, deep and lasting impact. Ultimately, it will change the way we think about business and the way that business decisions are made. Companies need to respond more quickly, gain operational efficiencies and deliver superior customer service by keeping a real-time pulse of their business. Real-time BI will play an essential role in tomorrow’s faster and more competitive business environment. Designing a suitable real-time BI tool to extract the latest information and support it with a decision system is not a cakewalk. However, researchers and various vendors all across the world are deeply engrossed in designing such solutions. Their studies and development of new approaches will bring fruitful results and can make real-time BI easier to implement and a household name among companies.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access