In Praise of the Smaller BI Vendors

The BI Survey 8 reveals surprising results for products and support

Information Management Magazine, Nov/Dec 2009

Nigel Pendse

The consolidation wave has been the big news in the business intelligence industry in recent years. First, there was the surge of acquisitions of smaller players by the larger specialist BI vendors, particularly Business Objects and Hyperion Solutions. Then the three largest specialist BI vfendors – Business Objects, Cognos and Hyperion – were themselves acquired by software titans from outside the BI industry. Now, SAP, IBM, Oracle and Microsoft all have large BI businesses, and much effort has been devoted to analysis of their complex product roadmaps.

This focus on the consolidation at the top end of the BI market has led to an assumption by many commentators that the remaining small vendors are now an irrelevance as far as the corporate BI market is concerned. But, looking at it from the customer’s viewpoint, which size of vendor actually delivers the most satisfactory results? Using quantitative results from the recent BI Survey 8, we can analyze real-world user experiences in a rational, objective way.

The BI Survey 8 is the world’s largest independent survey of BI users, so its worldwide sample of thousands of users and consultants provides a perfect opportunity to see if this industry consolidation is benefiting customers. The questionnaire asked about usage of 39 products, and respondents could also choose “other BI product.” We classified the vendors of these 39 products into small, medium and large categories:

We can analyze the collective experiences of users of products from these three vendor groups:

Problems Encountered

One might assume that products from large vendors are more likely to be problem-free. The vendors have large, well-equipped product development teams, using professional methodologies. They typically have extended development cycles, with multiyear roadmaps and lengthy internal and customer beta test programs, so problems should be spotted well before the commercial release.

These companies are experienced developers of large software products, used for critical applications like databases, transaction applications and operating systems, so it might be expected that they extend the same degree of professionalism to their BI products. Indeed, their slow and bureaucratic release cycles are often attributed to this.

To test this, the BI Survey asks respondents whether they have encountered any serious problems; they can select up to three from a list of 14. Of these, seven are product-related, with the remainder being people- or data-related. It can be assumed that people and data problems have nothing to do with the size of the vendor, so Figure 2 looks only at the complaint rates for product-related problems.

Clearly, confounding the expectations, users of products from smaller vendors are much less likely to complain of product problems than those using products from medium and large vendors. Indeed, with the most commonly reported problem (slow query performance), products from large vendors attract more than twice as many complaints as those from small vendors.

Even on the two scalability measures, products from small vendors did about twice as well as those from large vendors. In most cases, the products from medium-sized vendors also did better than those from the large vendors.

So, if their products are disappointing, do the large vendors make up for it with the excellence of their product support?

Product Support

The BI Survey asks respondents to rate the quality of product support they receive from their suppliers. They can choose from five levels, ranging from excellent – accurate and timely down to unacceptably bad. Figure 3 shows the mix for the three vendor sizes.

Again, the large vendors put in a very disappointing performance. Not only are their products more likely to attract complaints, but their subsequent product support also fails to meet expectations (let alone exceeding them, as per so many lofty mission statements). Only a shocking 13.4 percent of their customers are delighted with the accuracy and speed of product support from large vendors, compared to a rather more impressive figure of 41.5 percent for small vendors.

Fortunately, only a few customers complained of unacceptably bad support, but again it’s the customers of the largest vendors who had a poor experience, with 3.8 percent reporting it - almost three times the rate reported for small vendors.

The BI Survey also calculated a weighted overall support score from these ratings and found that the seven products with the worst scores are from four of the largest vendors, while the eight products with the best support scores come from small or medium sized vendors.

So if products from large vendors are more problematic, and are supported less well, do they at least redeem themselves by delivering higher levels of business benefit? After all, they are often chosen for strategic reasons that should be based on maximizing business benefits achieved.

Business Benefits Achieved

The whole purpose of any BI project is to deliver business benefits to the organization. This transcends any product issues, and so The BI Surveys use the level of business benefits achieved as a standard calibration tool for as many aspects as possible.

Page 1 of 3.