Continue in 2 seconds

The "Soft Side" of Real-Time BI

  • August 01 2004, 1:00am EDT

Although much has been written about the architecture and technology that drives real-time decision-making systems, there has been less discussion about the organizational and cultural issues involved in delivering a successful real-time environment.

There are several critical factors involved in real-time BI systems that have nothing to do with technology. For example, it's imperative that you understand how your businesspeople define real time and what they are willing to pay to reduce the latency of certain types of information. Your organization needs to restructure business processes to ensure real-time decision-makers are empowered to act on the real-time information, not just analyze it. Finally, users need to understand the limits of real-time data in terms of cleanliness and latency. A "Buyer beware!" sign needs to accompany the delivery of real-time data in the data warehouse.

Perception is Reality

So far, we've used the term real time in a general way without defining it. Technically, real time means something that happens instantaneously. In the world of data warehousing, this is impossible. There is always latency between the time an event occurs and when data is shipped to the data warehouse and made available to users.

In an effort to be more accurate, some people use the term near-real time to describe low-latency BI systems. But how fast is near-real time? Is it a second? A minute? An hour? In the end, the definition of real time or near-real time depends on the person to whom you are talking.

For example, airline executives interpret real time as anything that happens within 14 minutes, says Alicia Acebo, data warehousing manager at Continental Airlines. That's because the airline industry defines "on time" as 14 minutes or less from an aircraft's scheduled departure or arrival time. Several years ago, Acebo was pitching executives on a near-real time data warehouse that could capture and display business transactions within seconds. "They thought that near-real time meant it would take us more than 15 minutes to get transactions into the data warehouse," says Acebo. "Ever since, I've used their definition of near-real time."

On the whole, most people seem to agree that the term real time is a placeholder for "doing things faster." Because most organizations now update their data warehouses daily, TDWI defines real time to mean intra-day processing. That is, a real-time BI system captures transactions or event data that is distributed to users in the span of a few seconds to 24 hours.

Real Time to Right Time

Perhaps a better term than real time to describe what users want from their next-generation data warehouse is right time. Ultimately, business executives don't care about the degree of latency in a BI system. They simply want these systems to deliver the right information to the right people at the right time so they can make optimal business decisions. Right time puts the emphasis on the business value of information, not its latency.

Understanding Latency

One way to understand a right-time BI system is to analyze the types of latency that occur as information is moved from source systems to end users. Richard Hackathorn, president of Bolder Technology, offers an interesting framework that depicts three steps that introduce latency into the decision-making process and affect the business value of information (see Figure 1).

Figure 1: Framework for Right-Time Business Intelligence (Courtesy: Bolder Technology)
Organizations must manage three distinct processes that create latency in an analytic environment to support right-time decision making.

The objective of a right-time BI system is to reduce action distance so that users can respond to events in a timely fashion, says Hackathorn. Organizations can do this by addressing three types of latency depicted in Figure 1. To reduce data latency, for example, organizations can implement or reconfigure messaging middleware and data integration tools. To reduce analysis latency, organizations can rearchitect their data warehousing and information delivery systems.

However, decision latency is not driven by technology; it's a people issue. As the last leg in the loop, the business user can be the biggest bottleneck in the process. Business users need to understand the information, know what to do about it and be empowered to act on it. To address these issues, organizations need to rethink business process and organizational policies and provide additional training where required. Unless organizations reengineer underlying processes, they can't exploit real-time data in a cost-effective manner.

In some cases, decision latency can be eliminated altogether by using rules-driven processes that take action on behalf of the user or organization. Many organizations are using these so-called "agents" or rules-driven systems to automate actions in well-known business processes. Examples are automated fraud detection and credit rating systems.

Interestingly, some organizations compress data latency for reasons that have nothing to do with delivering right-time information or supporting operational decision making. Some organizations update their data warehouses in real time as transactions occur because they do not have a big enough window at night or on the weekend to process the transactions in batch.

"We process data 24 hours a day so we don't have to do it all at once," says Wes Flores, senior technology manager at Verizon. "This not only eliminates our processing window, it reduces our costs because we don't need as much hardware to handle heavy batch processing loads that only occur once a day or week."

Organizations that adapt to real-time data capture usually have fast-growing data warehousing environments. Either the volume of data or number of users is escalating rapidly. Global data warehouses that must be up and running 24x7 also have real-time data acquisition requirements even if they aren't yet supporting operational decisions. Whatever their motivation, these organizations are well positioned to support real-time BI when their users finally request it.

Making the Business Case

According to data warehousing professionals surveyed by TDWI, the chief obstacle is making the business case for real-time BI. Other challenges include nonintegrated data sources, lack of infrastructure, and education of business users, according to research from TDWI (see Figure 2).

Figure 2: Obstacles to Real-Time BI (From "The Real Time Enterprise Report," TDWI, 2003.)
Making a business case for real-time BI leads the list of obstacles, based on 383 respondents that have deployed a real-time BI system.

Real-time systems increase systems requirements and costs. The systems require more processing power and additional software, services and support. Trying to justify this additional overhead by touting the value of more timely data can be challenging for technical professionals. For many, it is a chicken-and-egg scenario.

"We need to prove the ROI of moving more of our system to a real-time environment, but we can't prove it until it's in place," says a data warehousing manager at an e-commerce company (who wishes to remain anonymous). "Right now, we're going after low-hanging fruit. We'll eventually get there, but it will be a long journey."

The aforementioned manager said he wished the organization had built real-time capabilities into the data warehousing infrastructure from the beginning, even if the business was not yet demanding it. This way, real-time capabilities could have been buried within the initial data warehousing infrastructure at little extra cost. This is exactly what Acebo at Continental Airlines did, and it proved to be an extremely wise decision.

"We built real-time capabilities into our data warehouse from day one because I knew users would eventually ask for it," says Acebo, who spent much of her career building online reservation systems. Specifically, Acebo and her team wrapped a batch extract and load process around Continental's message queuing infrastructure that transfers data among the firm's operational systems. Then they waited.

"After the terrorist attacks on September 11 in 2001, our management was ready to move from daily to hourly feeds of the data warehouse. The whole process took us one week," says Acebo. Subsequently, Continental received an award from the FBI for its role in helping track down terrorists because of the timeliness of information it provided the agency.

Sometimes organizations can bypass ROI exercises if they have visionary executives that intuitively understand the value of providing more timely data. "We didn't have to perform a ROI because all our executives know this is where we need to go," says Ken Kirchner, data warehousing manager at Werner Enterprises, Inc., a transportation company based in Omaha, Nebraska. "And our biggest backer is the company's COO."

In other organizations, the situation may not be as clear-cut. Even though the business may be asking for real-time data and the data warehousing team wants to deliver it, it's still wise to crunch the numbers. "We work with the business, tell them what it would cost to build a real-time system and let them decide whether it's worth it to them," says Verizon's Flores. "So far, the answer has been no."

Rethink Business Processes

The cost of real-time BI goes well beyond the dollars and cents required to purchase more hardware, software, services and support. Companies must reengineer business processes to exploit real-time data. For example, it is not useful to provide store managers with hourly sales data if they can only change prices or store displays once a week.

In some cases, the systems are simply catching up with business processes that have been working in real time but without adequate data. For example, the real-time BI system at Werner Enterprises will consolidate information about truck movements that managers now patch together from multiple screens in the transaction system, says Kirchner.

In other cases, the availability of real-time data will completely change the way the organizations do work. For example, the security department at Continental Airlines researched fraudulent activity by following tips they received from the field. Today, the department functions much more proactively and effectively thanks to the real-time BI system. The department now uses software "agents" that look for fraudulent activity by monitoring various data elements in the data warehouse. "The data warehouse has completely changed their processes and enabled them to recover millions of dollars in fraud," says Acebo.

Training and Data Quality

Sometimes the biggest bottleneck in the process is the business user. Organizations may need to retrain users to understand how to use and interpret real-time data. They may also need to create new policies so workers are incented to exploit the new information.

"Many of our users don't want the data to change underneath them," says Acebo. "We showed them how to create a report as of a certain time and date and turn off updates."

To foster real-time throughput, data warehousing managers often turn off referential integrity checks in the data warehouse database and bypass other validation and data cleansing programs, which can introduce errors into the system. Organizations need to tell users that real-time data hasn't gone through a formal reconciliation process and won't match numbers in the company's system of record at the end of the day.

At Werner Enterprises and Pfizer, because the target users already work with real-time data, they understand the nuances and limitations of the data. "Our finance clients already understand the dirt in the data so it's not a big deal," says Pfizer's Siegel.

Oddly, migrating to a real-time environment has improved the quality of some data at Continental. "When we close the doors on a flight, we immediately know if we are missing data and can call the station and get the answers. Previously, since we didn't cull information for our daily operations reports until the middle of the night, there was no way to retrieve the missing information," says Acebo.

Like it or not, data warehousing is no longer a batch-oriented decision support system designed to support strategic decision making. Employees at all levels of the organization - especially people in customer-facing positions - need to make operational decisions that require access to integrated information, the hallmark of a data warehouse.

To succeed in the world of real-time decision making, you need to understand how your businesspeople define real time and what real-time data is worth to them. This cost-justification process helps weed out the nice-to-have requirements from the need-to-have ones. You also need to empower decision-makers to act on real-time information, not just analyze it. This often requires restructuring business processes and roles. Finally, users need to understand that real-time data may not be as clean and consistent as other data in the warehouse.

Armed with this knowledge, your real-time BI efforts will have a much better chance of success. Good luck!

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access