Over the last couple of years, most of the business intelligence (BI) vendors have relabeled their products as performance management solutions. Their definitions of performance management have varied, depending on what products they happen to include in their suites, but they usually fail to demonstrate exactly how their products have helped the business performance of their customers. Even if they tried, it would be very difficult for them to show how the particular features of their products generated superior business performance compared to competitive products.With access to the data from The OLAP Surveys, such an analysis is possible, and the results are surprising.

The latest OLAP Survey had an even larger sample than the four previous editions: data from 2,100 participants in 95 countries was used to analyze dozens of aspects of the purchase and use of more than two dozen OLAP products. In total, 5,551 people responded, but some were not yet users, others were vendors using their own products and some were unable to answer questions. This is by far the largest independent survey of BI users ever carried out.

How Do You Measure Business Performance?

Because of the wide range of products included, The OLAP Surveys don't compare features of products - how could you compare the features of Hyperion Essbase and Oracle Discoverer? Instead, since the second edition of The OLAP Surveys in 2002, we've used a benefits-based benchmark. After all, why would you buy any BI product if you weren't hoping to gain business benefit?

We came up with a basket of eight possible business benefits. They cover various areas, focusing on aspects such as reduced costs, increased revenues, improved customer satisfaction and greater internal efficiency. We also allowed respondents to add a ninth if they thought it relevant. Few did, and because no consistent ninth benefit has emerged, we continue to use the same standard list of eight benefits:

  1. Saved headcount in IT,
  2. Saved headcount in business departments,
  3. Reduced external IT costs (hardware, support, consulting or software licensing),
  4. Saved other non-IT costs (e.g., inventory, waste, financing),
  5. Faster or more accurate reporting,
  6. Increased revenues through better sales and marketing analysis,
  7. Improved customer satisfaction through enhanced product quality and/or service levels, and
  8. Better business decisions through more thorough or timely analysis.

We have consistently found that "soft" benefits, such as improved reporting and better decision-making, are much more likely to be achieved than "hard" benefits such as headcount reduction, which throws doubt on claims of greater productivity resulting from the use of BI tools. We asked respondents whether each benefit was:

  1. Proven and quantified,
  2. Proven but not measured,
  3. Formally claimed but not verified,
  4. Informally suspected,
  5. Not achieved,
  6. Got worse/more expensive, or
  7. Don't know.

From these responses, we calculated a weighted score for each benefit, and then combined them into a composite business benefit index (BBI) - in other words, a key performance indicator (KPI) of how business performance has been impacted by the use of the product. Figure 1 shows the trends over the last four years.

Figure 1: Trends in Reported Benefit

Cynics might point out that better reporting, the most often achieved benefit, should be expected of any reporting application - which is what most BI applications are - and that this alone is not actually a business benefit. It's how you use the faster or more accurate reports that matters. However, it's outside the scope of this project to measure the competence of management rather than the success of BI projects themselves.

What shows up is that the top three and bottom two benefits have held the same positions for the last three years, with only minor switches in the mid-ranking benefits (see Figure 1). For all the claims that self-service reporting will reduce IT burdens, few companies have saved IT heads because of BI applications, though at least the trend seems to be in the right direction.

What Affects Business Performance?

The OLAP Survey uses the business benefit index to measure the effect of numerous factors, including:

  • Product choice,
  • Methods of product selection,
  • Architecture (MOLAP versus ROLAP),
  • Input data volumes,
  • Server platform,
  • License fees paid,
  • External consulting costs,
  • Type of lead implementer,
  • The extent of Web deployment,
  • Deployment time,
  • Data load/aggregation time, and
  • Query performance.

Figure 2 shows that of all the aspects evaluated, query time turned out to have the biggest impact on the business benefits achieved. Others, such as data volumes and server platform, had remarkably little impact on the BBI. This year, even product choice had little impact, though it was higher than in previous years.

Figure 2: Highest and Lowest BBI Values Across Eight Dimensions

In application terms, performance is a measure of how quickly something can be achieved. We therefore plotted the BBI against three reported time-related factors:

  • Deployment time (from purchase of software to initial deployment to users),
  • Latency time (time from when new data is available in source systems to when it is available for query in the BI application), and
  • Typical query times seen by end users in the largest applications.

Deployment Time

We all know that the longer it takes to deploy a project, the more that can go wrong: the business requirements get out of date, users lose interest, budgets get reassigned and so forth. From that follows the regular advice to "think big, but start small." While this may seem obvious, few people actually quantify the effect. We set out to do so in The OLAP Surveys, and the result is very striking (see Figure 3).

Figure 3: The Effect of Deployment Time on BBI

BBI peaks with deployment times of one to three months and then declines steadily and very predictably. The peak is probably because projects that can be deployed within a month, even if successful, are likely to be too small to deliver serious business benefits. However, when they take longer than three months, the risk of failure grows faster than the rise in potential business benefits from more ambitious projects.

The lesson is clear: simplify projects as needed to get the initial application into daily use within three months. Once users have had a chance to use it, they are almost certain to change their minds about what enhancements are needed. This revised set of user requirements should guide the next phase of the project (which should also be shorter than three months).


Some BI and ETL vendors are keen to promote their real-time BI capabilities. Most don't actually mean true real time, which would involve analyzing transactions as soon as they were generated, with no delays at all. A Wall Street trader might need this functionality, but most other applications in most businesses do not. What BI vendors actually mean by real time is better described as reduced latency.

With reduced latency, instead of only being able to see yesterday's or last week's data, the reporting application's database may be refreshed several times a day or even every few minutes. But reducing latency usually has a cost; it may require more processing power, more sophisticated software, more tuning or may require the users to tolerate slower queries. The obvious question is to ask to what extent the reduced latency increases business benefits. Again, The OLAP Survey 5 provides the answer for the first time (see Figure 4).

Figure 4: The Effect of Data Latency on the BBI

How much reduced latency is normally worth paying for? One hour. Latency times of less than this seem to have no consistent effect on BBI, but with longer times, it declines. There also seems to be a plateau between 10 and 30 hours, and it then declines again. In other words, latency of more than one day has a negative consequence. But it's also clear that latency has less impact overall on BBI than deployment time - so you can resist some of those demands for real-time BI with a clear conscience.

Of course, if your BI application is for Wall Street trading or running an oil refinery, it may be worth paying whatever it takes to deliver minimal latency times. For most of the rest of us, optimizing this factor turns out to be of limited value.

Query Time

Query time is the easiest to understand: it's the time elapsed from the user hitting the Enter key or OK button and seeing the report on the screen. It's not the artificial time that vendors quote in their boastful benchmarks or APB-1 tests that only measure the server component of response time. The users don't care about that - they are interested in the delays that they see, which include network delays and local PC processing time.

I've always known that fast query performance was important. I even included it in my FASMI (fast analysis of shared multidimensional information) definition in 1995. But until now, I've never found hard evidence of just how directly it impacts the overall business value of the application.

Figure 5: The Effect of Query Time on the BBI

Figure 5 is remarkable. Most people would expect that query performances of under one second were hardly different from query performances between one and five seconds, and there would be no impact on the overall business benefits achieved - but there actually is a measurable impact on business performance as query performance declines.

When compared to other factors, variations in query performance have more effect on the BBI than any other. If there is one aspect of an application that you want to optimize, choose this one.

Supported by a wealth of real-world data, the BBI is a valuable way of ranking factors that have an effect on the overall business value of an application. It's a way to estimate the effect the BI application has on business performance.

Many factors affect the BBI, but two with the greatest impact are the deployment and query times, which are often not given enough emphasis in BI deployments. At the same time, the much-hyped real-time BI turns out to have only limited value. For more information, see www.survey.com/olap.

Nigel Pendse is an independent industry analyst specializing in OLAP. Pendse is the lead author of "The OLAP Report" as well as "The OLAP Surveys."

This article originally appeared on www.dmreview.com.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access