Continue in 2 seconds

Enterprise Data Warehouses and the Network Effect

  • February 01 2001, 1:00am EST

Think of a Data Warehouse as an Online Community

The exciting new digital economy has highlighted an aspect of systems commonly called the "network effect." Basically, this is the recognition that in a network or online community, possible transactions (thus, potential value) can be realized through the interaction of almost any combination of nodes (people, links) in the system. Therefore, each additional person who joins an online community such as eBay or AOL raises the number of possible interactions (transactions) in the network by a multiplicative rather than an arithmetic factor. Conversely, each additional customer that a company such as Ford is able to attract adds value in a predictable and linear manner. Because network- based businesses such as eBay are able to attract new customers largely on the value (i.e., size) of their existing network or community, adding customers has a momentum-building impact that goes far beyond anything realized in a traditional business.

Assuming that a company has a large enough customer base to remain viable and provide a competent level of service and support, the quality of the customer experience in a traditional line of business does not directly correlate with the size of the customer base. Buying an automobile or an appliance from the number two or three supplier can be a fully satisfactory experience, and the customer is not necessarily missing anything that the customer of the market leader has enjoyed. Conversely, the quality of the customer experience for a company such as eBay is indeed a factor of the number of people in the network. Using the second or third most popular auction site means that your shopping experience is indeed less rich (i.e., more limited due to less possible interactions) than that which you would have enjoyed by using the market leader. As networks grow, they become ever more valuable.

While companies such as eBay and many of the Internet commodity and parts exchanges are built upon the network model, the power of the "network effect" can be exploited by companies in almost any line of business. One of the most valuable ways to do so is through the development and harvesting of information in an enterprise data warehouse (EDW). Think of the subject areas in your decision support environment as analogous to the nodes in a network, and we'll explore how the strategy behind your implementation (namely, whether to adopt a data mart or enterprise data warehouse approach) will have everything to do with how much total business value will be realized.

First, a quick review of the difference between the two approaches. Data marts are decision support environments that are built for a specific purpose, usually of fairly narrow scope. Frequently they are composed of a single subject area (e.g., finance, HR, sales and marketing, supplier, etc.) and are focused on business people in a single department. Because of their narrow scope, they can often be deployed rather quickly. On the other hand, enterprise data warehouses seek to pull all major subject areas onto a single platform. They are designed for widespread use across an organization and often take months or years to build to completion.

Multiplying Your Return on Investment

The network effect first came to my attention (though I did not know what to call it at the time) about five years ago when I was in the middle of a two-and-one- half year project to construct an EDW for a major transportation company. We populated six major subject areas, each for a different sponsor within the business, and garnered a significant return on investment merely as a result of enabling quick and easy access to the data that each user group had championed. Then, as people recognized the breadth of information that we had available on a single platform, demand quickly developed for analysts in department A to have access to data that had originally been populated for departments B, C and D. What should have been obvious from the very beginning of the project was that as valuable as any one subject area might be, it almost always had enhanced value when combined or viewed in conjunction with data from other subject areas.

To cite just one example of many, I would like to briefly discuss our cost subject area, originally created for the branch of our corporate finance department responsible for cost accounting. Soon after the data was populated, many other business groups clamored for access. Each had a story about the significant impact that the cost data would have on their own departmental operations when linked to their own subject areas. As a result, we quickly moved to link cost data to a host of transportation events in order to provide people in operations with both summary and detailed information about how operational events (e.g., reroutings, missed connections, fuel purchases, crew changes, etc.) impacted the cost of shipping goods. For our corporate auditors, we linked cost data to employee information to allow for analysis of travel and entertainment expense by employee, work group, department or vice president. Linking costs to employee training history identified cases in which poor or nonexistent training could be correlated with higher operational expenses due to mistakes, safety breaches, accidents resulting in lost work hours, etc. Last but not least, linking cost to revenue data provided for a view of profitability with a level of history and granularity that existed nowhere else in the enterprise (and certainly not in the poorly integrated transactional systems).

In the end, having cost data linked to revenue, operations and HR data within our EDW facilitated dramatically enhanced analysis and control of expenses for every facet of corporate activity. This data provided so much value beyond that of the standalone cost subject area that its use became ingrained within the daily processes of every major business department, and it was difficult to comprehend how we had gotten along before its advent. Had we implemented a standalone data mart, we would never have realized these benefits. It is easy, in retrospect, to see how cost data correlated at a low level of detail to information in other subject areas would be of great significance and business value. Other high-value linkages were less obvious. For instance, the intersection of fuel consumption, HR and training records provided a fuel efficiency rating for a critical part of the workforce and indicated whether lack of training may have been a factor for those with a below-average rating.

One of the consistently interesting aspects of EDW development is the creativity of business people in suggesting new and sometimes previously unimagined correlations between subject areas. Often these linkages form the basis for some of the greatest return on investment that a data warehouse can provide. In another major corporate project, we blended data from over a dozen different transaction systems to create a single view of a customer that was much more complete and valuable than any other company source of information. This formed the basis for a host of modified business processes and improved decision making.

Transaction systems, even ones that are great at automating the core business processes that they were designed to support, are almost always poorly integrated with other transaction systems. Compounding the problem is that most of these systems reflect the business structures for which they were designed, and these tend to be functional or oriented around lines of business. For companies that want to get a view of profitability that cuts across all product lines or to get a single view of customer purchasing history that cuts across all sales channels, transactional systems are usually poorly equipped to help. An EDW provides the best (and frequently the only) opportunity to bring data from different transactional systems together and facilitate analysis and decision making based upon data blended from all major facets of a business.

Even the Best Data Marts Don't Compare

Having had a key role in the planning and implementation of two multiterabyte enterprise data warehouses, I can confidently say that about 75 percent of the total return on investment was derived from applications and analysis in which data from multiple subject areas was linked together. In no way do I mean to imply that this is easy. The fact that the source systems for these subject areas are typically integrated poorly or not at all means that there are often no shared key fields that facilitate correlation. Thus, it sometimes requires a great deal of work to pull such data together. There are, however, few ways in which IT resources can be better used to achieve great business impact.

There has been quite a bit of spirited debate over the last 10 years as to the relative merits of the data mart and EDW approaches. Although I am firmly in the camp that believes in the superiority of the latter, this is based not upon some technical or ideological foundation, but merely on the recognition that the EDW approach is the only practical way in which to enjoy the great value add and return on investment to be realized by blending data from many different subject areas. The data mart approach, whatever other merits it may embody (and I cannot think of many), misses three-quarters of the potential inherent in the data. This is not to say that data marts have no merit. I have seen many that repaid their cost of implementation and support many times over. But why limit the benefit? Data marts, by their very essence, retread the same sins of the transaction systems that feed them ­ by offering a provincial view of data that is not integrated across the enterprise. The enhanced decision making that they facilitate is one that is optimized locally. Business areas are often satisfied with the result because they are unaware of the missed opportunities and because a well-constructed data mart offers much better analysis and reporting than the typical transactional system.

Decision support that can truly transform your business is never going to come about from a data mart strategy. Instead, let your business take advantage of the "network effect" ­ by building an integrated enterprise data warehouse in which any subject area can potentially be linked to any other. You'll be amazed at your return on investment. In the end you'll even find that your total cost of ownership is less. But that's a topic for another article.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access