The consolidation wave has been the big news in the business intelligence industry in recent years. First, there was the surge of acquisitions of smaller players by the larger specialist BI vendors, particularly Business Objects and Hyperion Solutions. Then the three largest specialist BI vfendors – Business Objects, Cognos and Hyperion – were themselves acquired by software titans from outside the BI industry. Now, SAP, IBM, Oracle and Microsoft all have large BI businesses, and much effort has been devoted to analysis of their complex product roadmaps.

This focus on the consolidation at the top end of the BI market has led to an assumption by many commentators that the remaining small vendors are now an irrelevance as far as the corporate BI market is concerned. But, looking at it from the customer’s viewpoint, which size of vendor actually delivers the most satisfactory results? Using quantitative results from the recent BI Survey 8, we can analyze real-world user experiences in a rational, objective way.

The BI Survey 8 is the world’s largest independent survey of BI users, so its worldwide sample of thousands of users and consultants provides a perfect opportunity to see if this industry consolidation is benefiting customers. The questionnaire asked about usage of 39 products, and respondents could also choose “other BI product.” We classified the vendors of these 39 products into small, medium and large categories:

We can analyze the collective experiences of users of products from these three vendor groups:

Problems Encountered

One might assume that products from large vendors are more likely to be problem-free. The vendors have large, well-equipped product development teams, using professional methodologies. They typically have extended development cycles, with multiyear roadmaps and lengthy internal and customer beta test programs, so problems should be spotted well before the commercial release.

These companies are experienced developers of large software products, used for critical applications like databases, transaction applications and operating systems, so it might be expected that they extend the same degree of professionalism to their BI products. Indeed, their slow and bureaucratic release cycles are often attributed to this.

To test this, the BI Survey asks respondents whether they have encountered any serious problems; they can select up to three from a list of 14. Of these, seven are product-related, with the remainder being people- or data-related. It can be assumed that people and data problems have nothing to do with the size of the vendor, so Figure 2 looks only at the complaint rates for product-related problems.

Clearly, confounding the expectations, users of products from smaller vendors are much less likely to complain of product problems than those using products from medium and large vendors. Indeed, with the most commonly reported problem (slow query performance), products from large vendors attract more than twice as many complaints as those from small vendors.

Even on the two scalability measures, products from small vendors did about twice as well as those from large vendors. In most cases, the products from medium-sized vendors also did better than those from the large vendors.

So, if their products are disappointing, do the large vendors make up for it with the excellence of their product support?

Product Support

The BI Survey asks respondents to rate the quality of product support they receive from their suppliers. They can choose from five levels, ranging from excellent – accurate and timely down to unacceptably bad. Figure 3 shows the mix for the three vendor sizes.

Again, the large vendors put in a very disappointing performance. Not only are their products more likely to attract complaints, but their subsequent product support also fails to meet expectations (let alone exceeding them, as per so many lofty mission statements). Only a shocking 13.4 percent of their customers are delighted with the accuracy and speed of product support from large vendors, compared to a rather more impressive figure of 41.5 percent for small vendors.

Fortunately, only a few customers complained of unacceptably bad support, but again it’s the customers of the largest vendors who had a poor experience, with 3.8 percent reporting it - almost three times the rate reported for small vendors.

The BI Survey also calculated a weighted overall support score from these ratings and found that the seven products with the worst scores are from four of the largest vendors, while the eight products with the best support scores come from small or medium sized vendors.

So if products from large vendors are more problematic, and are supported less well, do they at least redeem themselves by delivering higher levels of business benefit? After all, they are often chosen for strategic reasons that should be based on maximizing business benefits achieved.

Business Benefits Achieved

The whole purpose of any BI project is to deliver business benefits to the organization. This transcends any product issues, and so The BI Surveys use the level of business benefits achieved as a standard calibration tool for as many aspects as possible.

Respondents are asked the extent to which eight potential business benefits have been achieved, plus they can also nominate a ninth one, though few do. The benefits listed in the questionnaire were:

  • Saved headcount in IT;
  • Saved headcount in business departments;
  • Reduced external IT costs (e.g., hardware, external support and consulting, or software licensing);
    Saved other non-IT costs (e.g. inventory, waste, financing);
  • Faster or more accurate reporting;
  • Increased revenues through better sales and marketing analysis;
  • Improved customer satisfaction through enhanced product quality and/or service levels;
  • Better business decisions through more thorough or timely analysis; or
  • Other (specify).

For each of the nine potential benefits, respondents were asked to indicate the level of benefits achieved, if any, with six levels available. The weighted scoring system, shown in Figure 3, is then used to derive a composite weighted score for each of the possible benefits, based on the level of benefit achieved. This weighted score is called the business benefits index (BBI).

Let’s first look at the overall percentages of customers of the three vendor size categories who reported the six levels of business benefits achieved, for all nine benefits collectively.

Very clearly, the two highest levels of business benefits achieved are reported by customers of small vendors, while the four lowest levels are least likely to be reported by customers of small vendors. In particular, they were more than three times less likely to report that benefit levels had declined after a BI project compared to customers of medium and large vendors.

Looking at the BBI (the overall weighted score), the small vendors are again clearly ahead, with medium-sized vendors scoring slightly better than the large vendors (see Figure 6).

So, not only do the large vendors’ BI products perform less well at the product level, and are less well supported, but they also deliver less business benefit to customers.

Product Longevity

One assumed benefit from buying from large, financially stable vendors is the confidence that the companies will survive. While this is likely to be true, the longevity of the vendor most definitely does not always translate into product survival, particularly for products that represent a tiny, peripheral part of the large vendor’s business. In fact, BI products owned by large vendors whose main business is not BI have had an alarmingly high failure rate compared to those that continue to be owned by small, specialist vendors. It is all too easy for a large vendor to allow unimportant products to fade away – the vendor knows it can always replace them if needed, simply by buying another specialist vendor.

Look, for example, at how Microsoft, Oracle and SAP casually abandoned in-house developed planning and budgeting applications, rather than fixing them. In each case, the customers were offered no satisfactory way forward. At the same time, there are small, independent providers of equivalent products that have remained in business, supporting customers, for decades.

Why Aren’t Large Vendors Delivering?

While the Survey ruthlessly highlights problems, it cannot diagnose the causes, so what follows must inevitably be speculation, based on many years of anecdotal evidence rather than rigorous statistical analysis.

Perhaps the main problem is that large vendors don’t really have a BI strategy as such. They see BI as being a way to improve the attractiveness of their core products and as a cross-selling opportunity, rather than a business in its own right. Integration, rather than innovation, becomes the development focus.

Rather than making their BI products better and easier to sell to discerning new customers, they make sales through bundling their new BI offerings with other, more mainstream, products. The assumption is that the brand will sell the product, which no longer has to compete on its own merits. This allows the R&D costs to be cut, which is necessary to recover the high costs of recent BI acquisitions.

BI products become part of a bigger portfolio offering, and sales are made to IT buyers, rather than the users. The people buying the products, as part of larger deals, probably don’t perform any sort of competitive analysis against other BI products, assuming they are all more or less the same. Not only is this not the case, but they may not even be selecting the most appropriate products from their favored large vendor.

In time, the large vendors’ previous dedicated BI sales groups fade away or disappear altogether. The BI products also no longer enjoy dedicated support resources, so support quality suffers. Instead of working full time on a single product family, support people have to spread themselves thinly across many more products. This may be cost-effective for the vendor, but it does not help the users – some of whom will be paying an increased fee for this poorer support, because large vendors typically have higher maintenance charges than small firms.

Small vendors enjoy none of these luxuries. Their products and support levels have to remain superior or their customers will become disloyal. If a new release is problematic – which can happen to any product – the vendor has no choice but to focus all available resources on fixing the problems. Everyone from the CEO downward will be committed to keeping the customers happy, something that certainly won’t happen if a BI product from a large vendor is unsatisfactory. Small vendors know they have to deliver customer satisfaction to stay in business, and they cannot hope to simply coast along behind a strong brand.

So, the lesson seems to be that you should select the best or most appropriate BI product, regardless of vendor size. Good products are likely to survive regardless of ownership, while mediocre or poor products will deliver unsatisfactory results and may not survive for long, regardless of the size and stability of the vendor.

For more information on The BI Survey 8, including a free preview download, see

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access