The business intelligence (BI) industry, like many others, has a generous quota of myths, for all the usual reasons. Vendors have powerful commercial incentives to highlight the successes of their best customers while downplaying the flops. Users and project sponsors, who may have extracted large funding, are much more likely to spread the word about their extraordinary or beneficial projects than to bore the world with accounts of the far more numerous humdrum efforts. And, unless the problem projects were disasters on a scale that can’t be covered up (much rarer in the BI than the enterprise resource planning world), they have little reason to complain loudly about failures that may be at least partly their own fault.

Financial and industry analysts also have their own reasons to focus on the revolutionary, the exciting or the spectacular, while largely ignoring the far more numerous pedestrian deployments. And even if they do stumble upon a disaster, there may be legal prohibitions on blowing the whistle. Journalists often don’t have the personal knowledge or the time to dig deep, and they have to rely on a cast of characters all with vested interests for their information.

This biased information gets repeated so often that even “experts” who should know better become believers. Even though their own experiences may be less exciting than the glorious myths, they would still rather be seen as inspired than cynical. After all, which consultant or project sponsor could sell a new project if accompanied by prominent health warnings?

The BI Survey collects enough detailed information from large numbers of customers and implementers - over 1,900 in the latest edition - to test these myths. It’s hardly surprising that most turn out not to be true. But the extent of the disparity may still be a revelation. This article only has space to outline a few of them.

Exploding Data Volumes?

It is a truism of the data warehousing industry that data volumes are exploding, and there certainly are some multiterabyte data warehouses. But does this also mean that the typical BI application is getting ever larger? Most people would assume so, but The BI Surveys have found a rather surprising result: data volumes are surprisingly small, and there is no sign of any growth.

Figure 1 shows the median input data volumes of the overall sample for the last five years, and for three prominent products that typically handle large, medium and small data volumes. The overall sample had an input data volume of just 4.9GB in 2007, which was less than the 5.9GB reported back in 2003. In none of the years has it exceeded 6GB.

MicroStrategy sites consistently report the largest data volumes by far, but there is no increase in median data volumes among MicroStrategy sites. In fact, the 2007 figure was the second-lowest in the five-year period. The BusinessObjects samples have fluctuated more widely, but the 2007 sample reported slightly smaller volumes than the 2005 sample. The Microsoft Analysis Services (AS) volumes have hovered around the 5GB mark, and in 2007 they were actually lower than in each of the two previous years.

Although there may indeed be a few more very large BI applications these days, the medians have hardly changed in the last five years and remain around 5GB of input data. In other words, the giant BI applications are just outliers, and the typical application today is no larger than five years ago. So the next time someone confidently asserts that typical BI applications are handling exponentially increasing data volumes, ask them how they can be so sure about this factoid.

BI for the Masses?

It is hard to find a large BI vendor that has not claimed to be delivering “BI for the masses” at some point in the last decade. This claim has been variously based on claims of ease of use, high performance, low software prices or automated report distribution. But is it really happening?

The BI Survey has a number of metrics that can be used to track the validity of this claim. Let’s look at the median number of deployed seats - that is, software licenses that are actually installed and used, as opposed to sitting on the shelf. Again, we use median values to avoid the distorting effect of a few outliers.

Figure 2 shows that the “typical” BI site only has 30 deployed seats, and only two products had medians of more than 100 seats. Of course, the average seat counts are much higher because the occasional very large sites push up the mean values.

Looking back, the seat count in 2004 (the first year when this metric was reported) was 25 seats. It grew to 30 in 2005 and has stayed locked at that level ever since. So, not only are typical BI applications deployed to remarkably few users, but there has been no growth in the last three years: another myth busted.

Enterprise BI?

The large BI vendors have been promoting “enterprise BI” even more enthusiastically than BI for the masses. So let’s analyze the percentage of employees who regularly use BI applications.

The BI Survey 7 asked, “What proportion of all your organization’s employees currently make regular use of a BI application based on product?” Note that this asks about all the applications based on a particular product; most large organizations use multiple BI products, so the overall percentage using all BI tools is likely to be a bit higher. But a median of just 8.7 percent is still remarkably low.

There is some variation by vertical market, as Figure 3 shows (only industries with more than 30 respondents are included). It’s not surprising that the widest deployments are in the retail, telco and insurance industries, as these are well-known BI enthusiasts, but the low levels in banking and pharmaceuticals are unexpected.

These low percentages question whether we are anywhere close to having the frequent enterprise BI deployments that are claimed by most of the larger vendors.

Big Is Beautiful?

Buyers, particularly in large organizations, tend to be nervous about investing in products from smaller suppliers. They value the scale, resources and expected longevity of large vendors, as well as the expected integration between the wider product ranges offered by large vendors. But does this natural caution pay off for BI buyers?

The BI Survey provides numerous measures of project success and customer satisfaction, and across the board, the smaller vendors usually did better than the large vendors. For example, the Survey tracks the key Business Benefits Index (BBI) across many dimensions. The best-performing four vendors are all small, while the worst nine are all large. Similarly, shelfware rates were highest with large vendors and lower with smaller vendors.

Let’s focus on one important indicator: the quality of product support. The BI Survey lets respondents select from five levels of support quality, and calculates a weighted score from the data (see Figure 4).

The first column is most striking: almost 40 percent of the customers of small vendors reported that they received excellent support, compared to just 15 percent of the customers of large vendors. Conversely, twice as many customers of large vendors complain of unacceptably bad support compared to small vendors.

In particular, it seems that the vendors who have been most involved in the frenetic bouts of BI vendor consolidation deliver the worst support, while the small, one-product vendors do best. Presumably, the constant disruption and cost-cutting during the integration of support organizations is the cause. It’s not just the software vendors, either: the specialist BI consultancies delivered much more successful projects than large, generalist consulting firms.

So, another myth is busted: it is clear that BI buyers should focus on the product’s capabilities, not the vendor’s size and reputation. Of course, due to all the M&A activities, finding an independent small BI vendor is a lot harder than it used to be.

Don’t just take all the assumptions about the BI industry on trust - you need real research data, not just anecdotes. Things are changing much more slowly than we’re led to believe, and bigger often isn’t better. Most BI applications still handle only small volumes of data and serve just a few dozen users. And by almost every measure, small BI software vendors and specialist consultants do a better job than the giants.

For more information on The BI Survey 7, see www.BI-Survey.com.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access