Data warehousing has served us well over the past 15 years. It has allowed us to avoid the tyranny of business-as-usual as our only option. It has transitioned our legacy systems into the new e-commerce world. It has been a stepping-stone into the next-generation architectures for enterprise systems. Basically, it has reduced risk and decision time in dealing with those unconventional, unanticipated and messy business situations.

Warehousing has historically evolved to compensate for the deficiencies in operational systems. Reporting and especially analysis were afterthoughts for those legacy systems. We knew how to run our businesses. We assumed that processes were certain, exceptions were rare and markets were predictable. How naive this seems, given the uncertainties of the new millennium!

Not Your Mother's Warehouse

A recent EMA survey sponsored by DM Review asked 100 companies about their experiences with data warehousing.1

Eighty-seven percent had one or more warehouses in operation, of which 18 percent were over a terabyte in size. Thirty percent expect their warehouses to more than double over the next three years.

The most important purpose for the data warehouse is "ad hoc analysis as business problems arise" with 75 percent rating this purpose as critical. Financial analysis (at 82 percent) is still the most important application area; however, marketing analysis (at 79 percent) and customer relationships (at 70 percent) are of next importance. About half the companies are supporting more than 100 users. Additionally, half are supporting "general clerical personnel" as a major user group.

Half of the warehouses incorporate e-commerce information from Web sites, along with cleansing and enhancing customer data. Over one-third are feeding external data from other Web sites, search services and news feeds into their warehouses. Forty-four percent are using packaged BI applications, of which 30 percent have real-time feedback into their operational systems.

The data warehouse is no longer your mother's warehouse. We have developed a richer, broader and comprehensive notion of data warehousing which was inconceivable to most a few years ago.

The data warehouse holds the crown jewels of the enterprise. A well-designed and operated data warehouse is the closest realization of true corporate knowledge ­ distilled and encapsulated ­ that we have. To steal the secrets of a company, one should steal the schema definition and a few choice aggregation tables from the warehouse.

Turning the Warehouse Inside Out

We could follow our primitive instincts and guard these crown jewels in the fortress data center. We could lock down its data with strict need-to-know access and build the firewalls high and strong to avert any threat, inside or outside.

Yet the demands of our global economy are driving us in the opposite direction. They are whispering to us, "Turn the warehouse inside out."

Instead of locking the warehouse deep inside the data center, we need to extend the warehouse beyond the enterprise. The data warehouse provides the basis (i.e., the playing field) for interacting with our customers, suppliers, distributors and any other business partners. It has become too important to be confined by the boundaries of the enterprise.

Beyond the Enterprise

In many ways, the warehouse is already being extended beyond the enterprise.

Every warehouse needs reference data such as currency codes, country codes, postal/ZIP codes, telephone area codes and SIC industry codes. These codes provide important glue that links internal and external objects.

The common assumption is that reference data is static and nonvolatile. Once set, it does not change. However, the world does change. There are new currencies, countries change names and telephone area codes proliferate. Often, warehouse architecture does not adequately maintain the integrity of reference data.

Spatial processing opens up numerous applications based on data containing street addresses, ZIP+4 codes and GPS coordinates. Using the Sagent warehouse, a major insurance company conducted an analysis on the distance from the ocean for property claims. The concern was that insurance premiums were incorrectly assessed on many homes given actual damage from hurricanes. The finding was that homes with more than 1,000 feet of the shoreline did not sustain greater damages and, hence, should not be assessed higher premiums.

Numerous CRM applications have enhanced customer information with neighborhood and household data from census demographics and consumer segmentation. By licensing syndicated data, a rich (and chilling) composite of consumer behavior can be compiled from automobile registrations, county property records, telephone directories and many other public records.

Regardless of consumer concern, many companies have staked their futures on this practice of enhancing customer information for both B2C and B2B.

In the value chain, end-to-end supply chain applications are extending the warehouse beyond the single enterprise. i2 Technologies has built their warehouse so that the BI components of TradeMatrix can model multiple enterprises. For a major manufacturer, the warehouse contains the information needed to coordinate among many suppliers. For an industry consortium, the warehouse contains the complex many-to-many mappings of manufacturers, suppliers and products. In particular, it is critical to B2B exchanges that the warehouse supports a common product catalog that is stable, flexible, up-to-date, granular and structured.

Finally, we have only started to tap the potential of incorporating Web content with products such as Business Objects WebConnect and SAP Business Information Collector. The coming years will bring surprising transformations in the role and architecture of the data warehouse. Are you ready?

EMA Take- Away

  • Warehouse owners need to think through the business implications of providing warehouse information to external partners and integrating external data from commercial content providers and the Web.
  • Warehouse owners need to upgrade their infrastructure to insure adequate security, performance and stability as their warehouses are extended.
  • Vendors should provide viable business solutions built upon robust warehouse technologies incorporating innovative uses of external information and flexible access to external parties.

1. Telephone interviews based on a random sample from the DM Review subscriber list.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access