Gauge Your Data Warehouse Maturity
Many of us have managed data warehousing projects for years. Some have delivered highly strategic systems that are treasured by users and valued by top executives. However, others have struggled to sustain interest and funding in their data warehouses even though users are crying out for better, more accurate information.
What separates successful from struggling solutions? How does your data warehousing initiative compare to others in the industry? What will it take to get your data warehouse to the next level?
Many data warehousing managers ask these questions today. Unfortunately, there are no quick or easy answers.1 However, to provide some guidance, TDWI has developed a data warehousing maturity model that you can use to benchmark your progress. The model provides a quick way for you to gauge where your data warehousing initiative is now and where it needs to go next.
The maturity model consists of six stages: prenatal, infant, child, teenager, adult and sage. Business value increases as the data warehouse moves through each successive stage. (See Figure 1.)
Figure 1: Data Warehousing Maturity Model
Business value increases with each successive stage. Most organizations are currently in the child and teenager stages.
The stages are defined by a number of characteristics, including scope, analytic structure, executive perceptions, types of analytics, stewardship, funding, technology platform, change management and administration (for which we borrow concepts from SEI's Capability Maturity Model). This article will address just a few of these characteristics.
Organizations evolve at different rates through these six stages, and each may exhibit characteristics of multiple stages at a given time. As such, no one should expect to move cleanly and precisely from one stage to the next; however, there are two pivotal points in the evolution of any data warehouse/business intelligence (DW/BI) initiative, represented in Figure 1 as the "gulf" and the "chasm." Many DW/BI initiatives stall at these points. They remain stuck with one foot in the past and another in the future, unable to make a clean leap beyond. As a result, they never fully reap the benefits in the successive stages.
The primary way to overcome these obstacles is to change executive perceptions. To cross the gulf, executives must recognize that DW/BI is more than just a management reporting system and that the spreadsheets and desktop databases they rely on to run the business are actually undermining productivity and effectiveness. To cross the chasm -- which is much more difficult -- executives must perceive the DW/BI environment as a mission-critical enterprise resource that they own, direct and fund (not the IT department).
However, we're now getting ahead of ourselves. Let me briefly describe each stage and its major characteristics.
Prenatal Stage: Management Reporting
Most established organizations have management reporting systems that generate a standard set of static reports, which are printed and distributed to large numbers of employees on a regular basis, usually weekly, monthly or quarterly. Because the reports are hand-coded against legacy systems (or an operational data store), the IT department can't respond rapidly to requests for custom reports. This erodes IT's credibility and frustrates users who need quick access to information to do their jobs. This inflexibility is particularly intolerable to business analysts whose job is to crunch numbers on behalf of executives and power users who know their way around corporate information systems. Taking matters into their own hands, these individuals circumvent IT by extracting data from source systems and loading it into spreadsheets or desktop databases. This gives rise to our next stage.
Infant Stage: Spreadmarts
Spreadmarts are spreadsheets or desktop databases that function as surrogate data marts. Each contains a unique set of data, metrics and rules that do not align with other spreadmarts, management reports or analytical systems. Because spreadsheets are so ubiquitous, cheap and easy to use, spreadmarts proliferate like weeds -- organizations have dozens, if not hundreds or thousands, of these pernicious analytic structures.2
Spreadmarts prevent the organization (or CEO) from getting a clear, consistent picture of the enterprise. However, eliminating spreadmarts is difficult because they offer high local control at extremely low costs, making it difficult for organizations to cross the gulf between stage one and two. In fact, spreadmart users must sacrifice greater degrees of control over their analytical structures until the final two stages, when new development processes and analytical services align local and enterprise interests. 3 (See Figure 2.)
Figure 2: Local Control vs. Enterprise Value
It is difficult to eliminate spreadmarts because they offer maximum local control at minimum cost. Only when organizations enter the final two stages do local control and enterprise value align and accelerate together.
Child Stage: Data Marts
In the child stage, departments recognize the need to empower all their knowledge-workers with timely information and insight, not just the business analysts and executives who primarily benefit from spreadmarts.
A data mart is a shared, analytic structure that generally supports a single application area, business process or department. The departmental team gathers information requirements and tailors each data mart to meet the needs of the members in its group. They then provide knowledge-workers with an interactive reporting tool (e.g., OLAP or ad hoc query tool or parameterized reports). The tool lets knowledge-workers drill down or across a dimensional structure to follow trends and gain a deeper insight into events driving the process or tasks they manage.
However, data marts often fall prey to the same problems that afflict spreadmarts. Each data mart supports unique definitions and rules and extracts data directly from source systems. These so-called "independent" data marts do a great job of supporting local needs; however, their data can't be aggregated to support cross-departmental analysis. What's needed is a mechanism to integrate data marts without jeopardizing local autonomy. That's the hallmark of the teenager stage.
Teenager Stage: Data Warehouses
After building their third data mart, most departments recognize the need to standardize definitions, rules and dimensions to avoid an integration nightmare down the road. Standardizing data marts can be done in a centralized or decentralized fashion using one of eight strategies.4 The most common strategy is to create a central data warehouse with logical dependent data marts running in the same database as the data warehouse. This type of data warehouse is commonly referred to as a hub-and-spoke data warehouse.
Interactive Reporting and Analysis
Unlike single-subject data marts, a data warehouse encourages deeper levels of analysis. That's because users can now submit queries across functional boundaries, such as finance and operations, and gain new insights not possible when data was confined to operational and analytical silos.
To better monitor cross-departmental processes and enterprise value chains, organizations deploy dashboarding applications that support alerting, drill-down paths to detailed reports, distributed queries to pull data from non-warehouse systems and more timely loading of the warehouse. These dashboarding applications enable organizations to extend the benefits of business intelligence to more individuals than just technically savvy power users. As a result, executives value DW/BI as a tactical way to improve process efficiency, empower more users with information and embrace fact-based decision making.
Adult Stage: The Enterprise Data Warehouse
Although a data warehouse delivers many new benefits, it doesn't solve the problem of analytic silos. Most organizations today have multiple data warehouses acquired through internal development, mergers or acquisitions. Like spreadmarts and independent data marts, divisional data warehouses contain overlapping and inconsistent data, creating barriers to the free flow of information within and between business groups and the processes they manage.
In the adult stage, organizations make a firm commitment to achieve a single version of the truth. Executives view data as a corporate asset that is as valuable as people, equipment and cash. They anoint one data warehouse as the system of record or build a new enterprise data warehouse (EDW) from scratch. This EDW serves as an integration machine that continuously consolidates all other analytic structures into itself. A flexible business intelligence layer finishes the job by integrating data in the EDW with external data that is impractical to load into the EDW for one reason or another (e.g., real-time data feeds or Web data). Some organizations that have an acquisitions-based growth strategy use an EDW and BI tools as a prime vehicle to integrate newly acquired organizations.
Stewardship and Scorecards
In the adult stage, the EDW serves as a strategic enterprise resource for integrating data and supporting mission-critical applications that drive the business. To manage this resource, executives establish a strong stewardship program. Executives assign businesspeople to own critical data elements and appoint committees at all levels to guide the development and expansion of the EDW resource. On the analytic side, the organization deploys cascading scorecards to align every worker and business process with corporate strategy. The scorecards often lay on top of existing dashboard applications, refining existing metrics and prioritizing initiatives and budgets to support strategic goals.
During the adult phase, investments in the data warehousing environment begin to pay off. The EDW benefits from economies of scale and a fast-track development process that churns out new mission critical applications rapidly. (See Figure 3.) In addition, users begin to find new and unexpected uses for the data warehousing environment that developers hadn't anticipated. This serendipity of scale further accelerates ROI.
Figure 3: ROI and Maturity
A data warehousing environment begins to pay off in terms of ROI in stages 4 and 5.
Sage Stage: BI Services
Once the data warehouse becomes a strategic enterprise resource that drives the business with an ever growing panoply of mission-critical applications, you may think your job is done, and it may well be! However, there are additional opportunities to increase the strategic value of your EDW by driving the resource outward and downward.
Many companies today are already opening their data warehouses to customers and suppliers -- extending and integrating value chains across organizational boundaries and driving new market opportunities. Next-generation extranet applications will not just provide static reports on account activity. Rather, they will provide customers and suppliers with simple, yet powerful interactive reporting tools to compare and benchmark their activity and performance to other groups across a multitude of dimensions. Some companies, such as Owens & Minor, have created new business units to sell data warehousing and information analysis services, and they are transforming their industries as a result.
At the same time, EDW development teams are turning analytical data and BI functionality into Web services that developers -- both internal and external to the organization -- can leverage with proper authorization. The advent of BI services turns the EDW and its applications into a market-wide utility that can be readily embedded into any application. With BI services, workers will no longer have to shift contexts to analyze data. The data, information and insights they need to do their jobs will be embedded in the operational applications they use on a daily basis.
These BI services will also make it possible for companies to fully capitalize on their investments in statistical analysis and modeling. They will turn models into decision engines embedded in internal and external applications. Workers without any statistical background will feed information into these engines and receive recommendations instantaneously. Today, decision engines already form the basis of several types of powerful applications, including fraud detection, Web personalization and automated loan approval applications.
Once your DW enters the sage stage, its value increases exponentially as its visibility declines. As a BI service, the data warehouse and analytic server fade into the background, becoming critical infrastructure that no one thinks about until it stops working due to an outage. Our economy has commoditized innumerable services in the past, such as electricity, sewage, water and transportation. Insights delivered via BI services are simply next in line.
Whether you already exhibit the characteristics of a sage or you're still trying to hurdle the gulf between the infant and child stages, this maturity model can provide guidance and perspective as you continue your journey. The model can show you where you are, how far you've come and where you need to go. It provides guideposts to help keep you sane and calm amidst the chaos and strife we contend with each day.
1. TDWI has written several in-depth reports that can shed light on these issues. I would suggest reading TDWI's 2003 report titled, "Smart Companies in the 21st Century: The Secrets to Creating Successful Business Intelligence Solutions." This 40-page report uses quantitative analysis to shed light on what differentiates successful from struggling data warehousing and business intelligence solutions. You can download this report for free at http://www.tdwi.org/research.
2. Our latest research shows that organizations average 2.1 data warehouses, six "independent" data marts, 4.5 ODSs and 28.5 spreadmarts that they want to consolidate. From: "In Search of a Single Version of Truth: Strategies for Consolidating Analytic Silos," TDWI, July 2004. You can download the full 40-page report for free at http://www.tdwi.org/research.
3. For tips on controlling the proliferation of spreadmarts, see "Reeling in Spreadmarts," TDWI Case Studies and Solutions, June 2004.
4. Again, see TDWI's upcoming report, "In Search of a Single Version of Truth: Strategies to Consolidate Analytic Silos."