Benchmarks for BI and Data Warehousing Success

Information Management Magazine, January 2006

Hugh Watson, Thilini Ariyachandra

People and companies always want to know how they are doing. That's why they keep score, whether it is for golf or for business performance. The scores show how people and companies are doing over time, against goals or in comparison to others. Scores provide feedback and incentive to improve performance.

In business, benchmarks are especially useful. They are helpful in answering the question: How are we doing, especially in comparison to other companies? This is true in the business intelligence (BI) and data warehousing world. People want to know how successful their BI and data warehousing initiatives are in comparison to other companies.

We conducted a study (promoted by DM Review) that created metrics used for assessing the success of a data warehouse architecture and a company's use of BI. We collected survey data from 454 companies that can be used for benchmarking purposes.

In this article, we discuss how the success measures were selected, what the success metrics are, the survey data that was selected, the benchmark data for the metrics and the use of the metrics.

BI and Data Warehousing Success Metrics

Both practitioners and academics have a long-standing interest in measuring success. BI practitioners need to know how successful they are and communicate this assessment to management. Academics need to be able to measure success in their research.

To develop the success metrics, we reviewed practitioner and academic literature, and interviewed 20 leading authorities on BI and data warehousing. Two major success constructs emerged as relevant: product measures and development measures. Each of these constructs has component parts.

Product Measures

• Information quality: The data warehouse should provide accurate, complete and consistent information.
• System quality: The data warehouse should be flexible, scalable and able to integrate data.
• Individual impacts: Users should be able to quickly and easily access data; think about, ask questions, and explore issues in new ways; and improve their decision-making because of the data warehouse and BI.
• Organizational impacts: The data warehouse and BI should meet the business requirements; facilitate the use of BI; support the accomplishment of strategic business objectives; enable improvements in business processes; lead to high, quantifiable ROI; and improve communication and cooperation across organizational units.

Development Measures

• Development cost: The cost of developing and maintaining the data warehouse should be appropriate.
• Development time: The time to develop the initial version of the data warehouse should be appropriate.

These are not the only possible measures (e.g., system usage), but these were the ones most germane to our research.

Some of these measures relate specifically to the data warehouse, such as information quality and the warehouse development cost. Others, such as individual and organizational impacts, measure the success of both the data warehouse and the data access tools (e.g., Excel, Business Objects) and BI applications (e.g., EIS) that use the warehouse. Collectively, these measures provide a useful set of success metrics for the data warehouse and BI.

Data Collection and Analysis

Once again using the literature and experts, questions were developed for the success measures. In the survey instrument, respondents were asked to indicate (on a seven-point scale, with one being strongly disagree and seven strongly agree) the success of their BI and data warehousing initiatives. The job positions of the respondents were relatively evenly distributed over data warehouse managers, data warehouse staff members, IS managers and independent consultants/

system integrators. The latter were asked to complete the survey with one of their clients in mind. The companies included in the survey ranged from small (i.e., less than $10 million in revenues) to large (i.e., in excess of $10 billion). Most of the companies are located in the United States and represent a variety of industries, with financial services providing the most responses. The respondents are believed to be representative of the companies that use BI and data warehousing.

In addition to questions about the success metrics, respondents were asked to indicate their company's data warehouse architecture. Five choices were given:

1. Independent data marts,
2. Bus architecture with conformed dimensions (bus architecture),
3. Hub and spoke (i.e., Corporate Information Factory),
4. Centralized (i.e., no dependent data marts), and
5. Federated.

The independent data marts architecture scored lowest on the success metrics, followed by the federated architecture. Interestingly, the bus, hub-and-spoke and centralized architectures scored similarly on the success metrics for information quality, system quality, individual impacts and organizational impacts. Consequently, the scores were combined to provide benchmark numbers that can be used with any of the three architectures. Figure 1 lists the success metrics and the average scores based on the responses from 454 companies.

Figure 1: Benchmark Scores for Information Quality, System Quality, Individual Impacts and Organizational Impacts

The time and cost to develop the first business process(es) or subject area(s) varied across the three architectures. Consequently, they cannot be combined; separate benchmark numbers must be used with the bus, hub-and-spoke and centralized architectures. Figure 2 shows the benchmark numbers (median scores) for the time success metrics based on data from 348 firms and cost metrics based on data from 424 firms.

Figure 2: Benchmark Scores for the Cost and Time to Development

Most of the benchmark scores are 5.0 or higher. This indicates that, on average, companies are successful with their BI and data warehousing initiatives. The highest score is for users being able to access data more easily and quickly because of the data warehouse. Apparently, data warehouses are meeting their fundamental purpose of providing a repository of decision support data.

Page 1 of 2.