Continue in 2 seconds

Data Warehousing: The Past 10 Years Have Been Quite a Ride

  • February 01 2001, 1:00am EST

First, let me begin by congratulating DM Review on its 10th anniversary.

We have witnessed some significant changes and trends in the data warehousing industry during the past decade. Since DM Review's first issue, the magazine has consistently stayed ahead of industry trends, maintaining its position as the leading data warehousing and business intelligence publication in today's marketplace. More importantly, during this time I have had the pleasure of meeting and working with Ron, Jean, Juli and the rest of the DM Review's wonderful staff, and they don't come any nicer than these "guys."

Reminiscing on the Last 10 Years

As I think about the past 10 years, I am reminded of my first exposure to data warehousing. It was in the early 1990s when I was a partner at Price Waterhouse (PW), now Pricewaterhouse-Coopers (PwC). The firm decided to start a data warehousing practice, and I was asked to build and lead it. I was too young to know any better so I quickly accepted the challenge, even though my data warehousing experience at the time was somewhat limited.

Shortly thereafter, we held our first official data warehousing meeting which included about 20 partners and staff. Bill Inmon was invited to deliver a presentation on data warehousing; and, with his transparencies and drawings, he helped confirm our beliefs that data warehousing would become extremely important for our clients and our firm.

Reflecting the importance and growth of the entire data warehousing industry, today I am proud to report that PricewaterhouseCoopers has a data warehousing practice of more than 1,600 consultants representing every major country in the world. I also continue to appreciate and value my friendship with Bill.

In the last 10 years, we have experienced some important business and technology influences which have had a great impact on data warehousing. In the remainder of this article, I will touch on a few of the more significant trends.

Business Trends

The business environment for all industries and firms changed dramatically during the 1990s. Following is a chronology of the business trends that I believe had a pronounced impact on data warehousing. At the same time, data warehousing helped enable these trends.

Dynamic Business Environment

This term characterizes the 1990s. Company strategies were and continue to be focused on concepts such as: first to market, more customized products to more customers through more channels, new virtual business models, less government regulation, increased competition, changing workforce and others. Facing myriad challenges, companies have come to recognize the importance of having the right information to support their business strategies and market forces.

From building data warehouses, to extracting and consolidating data from mainframe-based legacy systems, to analyzing data from a company's Web site, the value associated with data warehousing and decision support was greater than ever during the past 10 years.

Industry Consolidation and Globalization

To address the dynamic business environment of the 1990s, many firms acquired or merged with other organizations. Typically, the primary objective of these mergers and acquisitions (M&As) was either to extend a company's product line or "footprint" (Citigroup, Cisco and others) or to expand geographically (Ford, GM and BP Amoco).

The process of planning and executing these M&A transactions required timely and accurate information. As these mergers and acquisitions occurred in the 1990s, the value of information began to extend beyond traditional financial analysis. Companies began to recognize that their business intelligence was a strategic asset with inherent value. As such, investment bankers started inquiring as to whether these companies had a data warehouse that included customer information which could be used to benefit the combined organization in areas such as cross- selling and placing a quantifiable value on this information.

Business Process Reengineering (BPR) and ERP

As we entered the mid-1990s, many companies were committed to transforming their outdated business processes and organizational models in order to more effectively manage costs and take advantage of new client/server technology. To achieve these objectives, and in anticipation of having to transition legacy applications to be Y2K-compliant, many companies purchased new enterprise resource planning (ERP) applications. These applications allowed companies to improve their processes for a single function (e.g., finance), while also providing integrated transaction processing across multiple functions (e.g., order management, billing, accounts receivable and others). These ERP investments also helped many firms avert potential Y2K disasters.

From a data warehousing perspective, this trend was a mixed blessing. On the positive side, the process changes helped "de- layer" many organizations by placing more personnel (knowledge workers) on the front lines to directly serve customers. At the same time, the new ERP solutions helped make these individuals' jobs easier by automating more of these transaction processes (i.e., data entry) and allowed companies to capture and store more transaction-level data.

On the other hand, the value of BPR and ERP was not fully realized in many cases. Companies were unable to provide the tools and information that knowledge workers needed to effectively fulfill their new roles. For instance, as a result of not having all relevant customer information, a bank lending officer may have denied a car loan application for a teenager, not realizing that the teenager's father was the chairman of the bank's largest institutional customer.

While many companies gained tremendous value from their ERP investments, there continues to be mounting frustration from mid- to upper- management because these applications do not provide them with the analytical capabilities they require. The fact of the matter is that these applications were never designed to provide online analytical processing (OLAP) functionality. As a result, data warehousing solutions have been successfully deployed to extend the value of ERP and other source systems, delivering the right level of information and analysis to knowledge workers and management alike.

Customer Centricity

During the past few years, we have seen companies of every shape and size move from being product- centric (we make it and you take it) to becoming customer-centric (understanding the customers and how they would like to be served).

In my opinion, to become customer centric and successfully achieve your customer relationship management (CRM) objectives, you must have a customer data warehouse as your foundation. Furthermore, the customer data warehouse must include all relevant information about a company's customers along with the required analytics to perform customer segmentation, churn analysis, propensity to buy, one-to-one marketing and other mission-critical CRM programs.

Many companies invested in disparate CRM applications in the 1990s ­ sales force automation, call-center technologies and marketing automation (including campaign management), to name a few. Very few of these companies have been able to successfully integrate the information from these applications to provide a "single view" of the customer.

To accomplish these customer-related strategic objectives, a company must have an integrated customer data warehouse as the "glue" for integrated CRM. Consequently, during the past two years we have seen CRM drive tremendous demand for data warehousing. In fact, I would submit that customer centricity has driven more demand for data warehousing than any other business trend during the past decade.


There's little question that the Internet represents the most disruptive technology ­ relative to its impact on how business is conducted ­ since the industrial revolution. Beginning with the business-to-consumer (B2C) model popularized by e-retailers (e.g., through business-to-business (B2B) models (e.g., e- marketplaces) to business-to-employee (B2E) initiatives that many firms are embracing to provide more personalized information to employees via portal technologies, the Internet is changing all of our lives.

I discussed the importance of data warehousing within each of these three e-business models in an article entitled "E-Analytics ­ The Next Generation of Data Warehousing," which was published in DM Review's August 2000 issue. In essence, the article describes the new economy as an information-driven economy. In this new economy, companies consider their information a strategic asset with the potential to provide real competitive advantage if the data can be aggregated and then fully leveraged using rich analytics.

In my view, the aforementioned e-business models must apply a data warehousing solution in order to be effectively and profitably implemented. The importance of data warehousing to e-business started with B2C and the CRM movement in 1998 and continues to grow today as the Internet becomes an increasingly important customer channel for most companies. At the same time, we are entering a dramatic growth period for B2B which will exponentially increase the availability of and demand for information, and thus data warehousing, at a rate unlike anything we have ever experienced. The companies that understand this and lead their industries into collecting and harvesting this information will have a huge advantage in turning B2B into a profitable business model. This is similar to what we saw with B2C, where many early stage e-retailers did not take advantage of their information assets. As a result, many of these companies are no longer in business.

These and other significant business trends have helped popularize data warehousing. It is interesting to note that during this ten- year period, data warehousing has gone from being a program typically owned and sponsored by IT to being a strategic imperative for business executives across the enterprise (to include such areas as finance, marketing, distribution/logistics, human resources, sales, customer service, manufacturing, etc.).

Technology Trends

In addition to the business trends mentioned, the past 10 years were filled with dramatic advances in technology. These advancements had a tremendous influence on data warehousing.

In the early 1990s, larger companies ran most of their technology applications on mainframes. A large percentage of these applications were custom built with the exception of finance, which had begun embracing packaged applications (i.e., general ledger, accounts payable, billing, etc.) during the 1980s.

Many of these companies were introduced to data warehousing as a solution for sourcing and integrating data from several mainframe legacy systems to enable aggregated reporting and some degree of analysis. During the early 1990s, several desktop OLAP tools were introduced. These tools provided users a friendlier way to interact with a segment of the warehouse data.

Soon thereafter, client/server computing came to the forefront as the new generation of information technology. In addition to triggering the ERP trend described previously, client/server technology was instrumental in advancing data warehousing architectures, as it enabled distributed computing. Many data warehouses and data marts leveraged this technology as a way to move more data closer to the desktop.

During the 1990s, distributed computing became the predominant architecture; most hardware and software companies focused their research and development efforts on developing new and enhanced products that were compatible with this new technology. Specific to data warehousing, we saw tremendous progress relative to both the functionality and scalability associated with products in extract/transform/load (ETL), data repositories/databases, OLAP, data mining and other associated decision-support technologies.

In the past few years, we have seen the rise of the Internet. The Internet's impact on data warehousing will be tremendous in terms of enabling more individuals to access and gain value from existing warehouses ­ beginning with intranets and, more recently, making the information available to trading partners via extranets. At the same time, the Internet provides valuable information about customers, suppliers and competitors that was not readily available from traditional sources.

A data warehouse is an integral part of Internet architectures. To be successful implementing an e-business strategy, companies are deploying data architectures that leverage data warehousing within a technical architecture that takes advantage of the relative strengths of each of the three major technological frameworks that have emerged during the past decade: mainframe, client/server and the Internet.

Recognizing the Efforts of Data Warehousing Professionals

While this article focuses on the business and technology trends we have witnessed during the past 10 years, this article would not be complete without a brief mention of the people who were instrumental in the advancements. From the "founding fathers" such as Bill Inmon and Ralph Kimball (and his team at Metaphor) to the modern day data warehousing enthusiasts, we have an outstanding group of professionals who have helped advance the state of data warehousing far beyond what could have been imagined just 10 years ago.

Most importantly, thousands of individuals around the world have dedicated their academic and/or professional careers to data warehousing and have worked endless hours to design and implement data warehousing solutions that have created tremendous value for companies of all shapes and sizes. Their work has been phenomenal and has made a tremendous contribution to the advancement of the entire data warehousing industry.

Envisioning the Future

It is easy to look back at the past 10 years and the growth of data warehousing with a sense of accomplishment and pride. However, if we shift from the past to the present, we see more companies adopting data warehouses in more strategic areas than ever before. If we look to the future, we see an accelerated growth as companies grapple with aggregating and analyzing data that is expected to double in volume every year.

So, while the past 10 years have been quite a ride for the data warehousing industry, hold on tight because the next 10 years promise to be even better.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access