It's rare enough to have a single unbiased, objective non-vendor-sponsored, statistically valid snapshot of what is really happening in the business intelligence (BI) industry. However, with the recent publication of The OLAP Survey 4, we now have a series of studies conducted using a consistent methodology, which lets us look at trends. In other words, this series lets us examine what hasn't changed as well as the direction of the things that are changing.
As with previous editions of The OLAP Survey, this fourth edition included the DM Review reader panel; therefore, some of you may have participated in the latest survey. However, this time we decided to try to obtain a more international flavor to the sample overall, so we teamed up with a German industry analyst organization to get a much clearer picture of the situation in German-speaking countries. The questionnaire was fully translated into German, and far more people in continental Europe were included in the study.
One interesting difference is that the response rate was higher from German respondents, probably because they are less affected by spam. Unsolicited English e-mails inviting participation in an activity that involves clicking on a link are far more likely to be spam-trapped or simply ignored by jaded recipients. Thus, among all the other burdens, spam must be added to the increasing difficulty of conducting genuine research.
Despite these difficulties, we managed to get responses from almost three thousand respondents in 48 countries, of which almost one thousand provided detailed information about their experiences in buying and using dozens of OLAP tools. Based on all this information and the trend data from the previous three editions, The OLAP Survey 4 has almost 300 pages of detailed analyses believed to be the most thorough ever conducted of the BI industry.
What Goes Wrong?
The OLAP Survey asked respondents which of many possible problems occurred; they could select as many as three that were the most serious. We subsequently classified these as people-, data- or product-related problems. In every edition of The OLAP Survey, the worst problems in the three categories have remained unchanged (see Figure 1):
- Worst people-related problem: company politics.
- Worst data-related problem: data quality.
- Worst product-related problem: query performance too slow.
Figure 1: Problem Trends
However, in this seemingly unchanging landscape, something remarkable has happened. Company politics and poor quality data are receding. They remain important inhibitors to project success, but there have been real improvements over the four years. However, despite all the hardware and software advances, query performance is becoming even more of a problem than it was previously.
It may be predictable that all the investments in ETL tools, data warehouse projects and modern ERP systems have paid off in the form of cleaner data for analysis, but the even sharper fall in the incidence of company politics is more of a mystery. Perhaps there is growing realism about what is possible in BI projects.
Despite the reduction in political problems, people-related problems as a whole still occur more often than technical problems. Overall, people-related problems seem to occur most often in medium-sized companies, though company politics increase steadily with organization size. What does get markedly better in the larger organizations is the ability to agree on requirements. Large organizations probably have a more formal methodology, having no doubt undertaken similar projects in the past.
German sites also reported relatively fewer people-related problems; but in this case, it was because they had less trouble with a lack of interest from business users.
Data problems vary surprisingly little with organization size or location. Even data volumes had little impact, with respondents with small applications reporting data volumes as a problem almost as often as those with the largest volumes. Curiously, however, we did find a significant variation by product (see Figure 2). This obviously has little to do with the products themselves, but is probably a reflection of how they are used.
Figure 2: Data Problems by Product
BusinessObjects users reported the fewest data access problems, perhaps because its universe concept simplifies data access compared to other products. As might be expected, SAP BW users also had less trouble getting data than users of most other products; this is, of course, a result of BW's close integration with SAP operational systems.
However, curiously, BW users actually reported more problems with data quality than did the users of any other product. This may be because data is typically loaded directly into BW from SAP R/3, rather than going through the traditional data integration and cleansing process that would happen in a full scale, full-function data warehouse.
Almost at the other extreme, and despite its typically larger data volumes, MicroStrategy users were among the least likely to report data quality problems. Again, this is not an attribute of the product, but might be an indication that MicroStrategy is typically used in conjunction with well-structured data warehouses. Oracle Express sites are typically very mature, and data quality problems were probably sorted out some time ago, which might explain why it is rarely a serious problem for them.
As in every previous OLAP Survey, query performance was the most often reported product-related problem, and, as in the past, ROLAP (relational online analytical processing) products (those that store the live data in a relational database) suffered more than MOLAPs (multidimensional online analytical processing), which use proprietary optimized cube databases. This was true even when the larger data volumes that ROLAPs typically handle were taken into account (see Figure 3).
Figure 3: Percentage of Sites Complaining of Poor Query Performance
The three products with the highest incidence of performance complaints were all ROLAPs, and the five with the fewest problems were all MOLAPs, with Applix TM1 users particularly happy about performance. However, one ROLAP product, MicroStrategy, has improved markedly over the years. In 2004, its performance complaint rate beat the average, and its actual query times, when its large data volumes were taken into account, were not far outside the speedy MOLAP band. Only two years earlier, in The OLAP Survey 2, approximately 34 percent of MicroStrategy users complained of performance problems compared to just 16 percent in this survey. This seems to demonstrate the benefit of the optional data caching now offered in MicroStrategy 7i. Conversely, PowerPlay was the only MOLAP to have performance slightly worse than the overall survey average, though this has also happened previously.
As mentioned in the performance comments, ROLAP products typically handle much larger data volumes than MOLAP products, but even within each category, there are wide variations. However, some very small applications use ROLAP architectures, and a few of the very large systems use MOLAP.
Data volumes also tend to rise with organization size and vary geographically (see Figure 4).
Figure 4: Median Input Data Volumes (GB)
In last year's article (please see: www.dmreview.com/article_sub.cfm?articleId=8174), I commented on how difficult people found it to predict their future deployments, and the trend has continued in The OLAP Survey 4. The OLAP Survey mainly focuses on questions about the present and the past rather than asking about future expectations; however, the two exceptions are plans for intranet and extranet deployments. Now, with four years of data, we can perform the comparisons between expectations and reality that so few forecasters do. We constantly read forecasts from analysts, gurus and user surveys, but very seldom do we see analyses of just how close to reality they were.
The chart in Figure 5 plots the percentage of sites that were, or expected to be, at least 50-percent Web-deployed. It shows that the real Web deployment rate has been consistently lower than predicted. Even looking only 12 months ahead, sites have been, on average, at least 35 percent over-optimistic, though it does appear as though the forecast errors are decreasing.
However, perhaps this is a slightly fuzzy target that is difficult to predict. Would forecasts of a more binary outcome be more accurate? To help assess this, we also asked about current and expected future extranet deployments.
Figure 5: Web-Deployment Trends vs. Forecasts
Deploying OLAP data over an extranet is a very specific, binary act that should be easier to predict because it is not a casual, last-minute activity. The security policy must be set well ahead, along with the technical architecture. Just as important are the decisions about who will get access and who funds the system. Building and testing the system is likely to take much longer than it would for most intranet applications. All this means that the plans for deploying OLAP data over an extranet should be decided long in advance. Are predictions any more accurate for extranets than for intranets?
As Figure 6 shows, the predictions were even worse for extranets than intranets. The real rate is hardly growing and remains stuck at approximately 15 percent; however, predictions from only a year earlier were for approximately 30 percent, and from two years earlier, close to 50 percent. Clearly, many people are wildly optimistic in their forecasts about the rate of exploitation of Web technologies in general, and extranets in particular. This means that typical published forecasts of widespread extranet deployment in the near future can probably be taken with a pinch of salt - and vendors relying on it for growth will probably be disappointed.
Figure 6: Extranet Deployments and Predictions
Why are forecasts so wide of the mark? After all, these are not forecasts from "interested parties" such as vendors or industry analysts keen to promote the idea that change is happening faster than it really is. These forecasts are from ordinary users with no reason to distort the truth. I speculate here, but it seems clear that these distortions are the result of an insidious feedback loop:
- Interested parties, such as people marketing new products or industry analysts trying to be newsworthy, have a vested interest in predicting the rapid take-up of a new technology.
- The press has a bias toward reporting predicted revolutionary changes; mundane "business as usual" or gradual change announcements are often more accurate, but are also less newsworthy.
- Ordinary consumers and businesses are therefore subjected to a barrage of propaganda from both the media and marketing campaigns about the wonderful world that lies just ahead if they only rapidly adopt the latest, over-hyped technology.
- Not wishing to be left behind, they duly plan to adopt the new wonder technologies.
- However, when it comes time to deploy, the products often turn out to be immature, the technology unproven, the costs higher than expected and the benefits elusive; therefore, real-world deployments fall well behind the predictions.
- By then, the technology industry has moved to a new set of extravagant predictions, and few people bother to comment on the current disappointments.
These documented shortfalls come from a specific area in BI, but I suspect that this cycle is repeated throughout the technology industry.
This review has highlighted some of the problems, but overall, the picture is positive this year. Projects are encountering fewer problems and delivering more benefits than in previous years. There is also some evidence that products are being chosen for more rational reasons.
For more information, see http://www.survey.com/olap.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access