Continue in 2 seconds

My, How Times Change! 10 Years of Data Warehousing

  • Claudia Imhoff, Jonathan G. Geiger
  • February 01 2001, 1:00am EST

You're probably familiar with the technology adoption curve. Companies first to adopt the new technology are called innovators. The next are known as the early adopters. Then, we move to the early majority, the late majority and, finally, the laggards. The curve is a traditional bell curve with exponential growth in the beginning and a slowdown in market growth occurring during the late majority period. When new technology is introduced, it is usually hard to get, expensive and imperfect. Over time, its availability, costs and features improve to the point where just about anyone can benefit from ownership. Cell phones are a good example of the technology adoption bell curve. Once, only the innovators carried them. The phones were big, heavy and expensive. Service was spotty at best, and users got "dropped" a lot. Now, there are deals where you can get a cell phone for approximately $60, the service providers throw in $25 of air time and there are no monthly fees.

Data warehouses are another good example of the adoption bell curve. In fact, if you haven't started your first data warehouse project, there has never been a better time. Executives today expect, and often receive, the good, timely information they need to make informed decisions and lead their companies into the next decade. This wasn't always the case.

Just a decade ago, these same executives sanctioned the development of executive information systems (EISs) to meet these same needs. The concept behind the EIS initiatives was sound ­ to provide executives with easily accessible key performance indicator information in a timely manner. Many of these systems fell short of their objectives, largely because the underlying architecture could not respond to the rapidly changing environment.

During the last 10 years, we have seen significant changes in the underlying architecture, tools, technology and techniques. These innovations have brought data warehousing to the forefront by helping companies improve their operations and relationships with their customers.


One of the major changes during the last ten years was the introduction of a widely accepted architecture to support executives' demands. This architecture recognized that the EIS approach had several major flaws, the most significant of which was that the EIS data structures were often fed directly from source systems. This resulted in very complex environments that required significant human and computer resources to maintain. The Corporate Information Factory (see Figure 1), which is a sustainable architecture used in today's decision support environment, addresses that deficiency by segregating data into four major constructs and incorporating processes to effectively and efficiently move data from the source to the business users as follows:

  • The operational system databases contain the data used to run the day-to-day business of the company. These are still the major source of data for the decision support environment.
  • The data warehouse is a collection of historical data to support strategic decisions.
  • The operational data store is a collection of current data to support tactical decisions.
  • The data marts are derivatives from the data warehouse used to provide the business community with access to various types of strategic analysis.

The Corporate Information Factory didn't just happen. In the beginning, it consisted of the data warehouse and sets of lightly and highly summarized data. The Corporate Information Factory was initially a collection of the data needed to support strategic decisions. Over time, it spawned the operational data store with a focus on the tactical decision support requirements. The lightly and highly summarized sets of data evolved into what we now know as data marts.

Figure 1: The Corporate Information Factory

Tools, Technology and Techniques

One of the major shortcomings of the early executive information systems was that they required enormous effort to provide the executives with access to the data they desired. Data acquisition or extraction, transformation and load (ETL) is a complex set of processes, and its sole purpose is to attain the most accurate data possible and make it accessible to the enterprise through the Corporate Information Factory. Data acquisition brings data from the operational systems into the data warehouse and ODS. The entire process began as a manually intensive set of activities. Hand-coded "data suckers" were the only means of getting data out of the operational systems for access by business analysts. This is similar to the early days of telephony when operators on skates had to connect your phone with the one you were calling by racing back and forth and manually plugging in the appropriate cords.

Fortunately, we have come a long way from those days, and the data warehouse industry has developed a plethora of tools and technologies to support the data acquisition process. Now, most of the process can be automated, much like the progress in today's telephony world. Also, similar to telephony advances, this process remains difficult, if not temperamental and complicated. No two companies will ever have the same data acquisition activities or even the same set of problems. However, the overall process is similar for all enterprises. Today, most major corporations with significant data warehousing efforts rely heavily on their ETL tools for design, construction and maintenance of their environments.

Another major change during the last decade was the introduction of tools and modeling techniques that bring the phrase "easy to use" to life. The dimensional modeling concepts developed by Ralph Kimball are largely responsible for the widespread use of data marts to support online analytical processing. Other sophisticated technologies have evolved to support data mining and exploration needs as well.

We shouldn't underestimate the impact of the Internet on data warehousing. The Internet helped remove the mystique from the computer. Executives use the Internet in their daily lives and are no longer wary of touching the keyboard. The end-user tool vendors recognized the impact of the Internet, and several of them capitalized on that impact by designing their interfaces so that they replicated some of the look-and-feel features of the popular Internet browsers and search engines. The sophistication ­ and simplicity ­ of these tools has led to a more widespread use of the data warehouse by business analysts and executives.


Another major change during the last few years was the switch from technology chasing the business to the business demanding the technology. In the early days of data warehousing, the information technology group recognized the value and tried to sell the merits of data warehousing to the business community. In some unfortunate cases, the IT folks set out to build a data warehouse with the hope that the business community would use it. The value of a sophisticated decision support environment is widely recognized today. An effective customer relationship management program could not exist without strategic (data warehouse with associated marts) and tactical (operational data store) decision-making capabilities (see Figure 2).

Figure 2: Strategic and Tactical Portions of the Corporate Information Factory

Customer relationship management (CRM) requires more than just the technology ­ it requires alignment of the business strategy, corporate culture and organization with the customer information and technology to provide long-term value to both the customer and the organization. The data warehousing architecture, such as that provided by the Corporate Information Factory, fits very well within the CRM environment.

  • The operational systems continue as the backbone of the enterprise to run the day- to-day business.
  • The data warehouse collects the integrated, historical data supporting customer analysis and segmentation. The data marts provide the business community with the capabilities to perform these analyses.
  • The operational data store supports the near real time capture of integrated customer information and the management of actions to provide personalized customer service.

CRM is a popular application of the data warehouse and operational data store, but there are many other applications. For example, enterprise resource planning (ERP) vendors such as SAP, Oracle and PeopleSoft have embraced data warehousing and augmented their tool suites to provide the needed capabilities. Many software vendors are now offering various plug-ins containing generic analytical applications such as profitability or campaign analysis.
The evolution of data warehousing has been critical in helping companies better serve their customers and improve their profitability; this evolution required a combination of technological changes and a sustainable architecture. The tools for building this environment have certainly come a long way. They are quite sophisticated and offer great benefit in the design, implementation, maintenance and access to critical corporate data. The Corporate Information Factory architecture capitalizes on these technology and tool innovations. It creates an environment that segregates data into four distinct stores, each of which has a key role in providing the business community with the right information at the right time, in the right place and in the right form. So, if you're a data warehousing late majority or even a laggard, take heart. It was worth the wait.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access