The widespread use of data warehousing techniques has only recently become part of the mainstream computing world. Although most people tend to think of data marts and data warehouses as relatively new phenomena, it turns out that there are actually a significant number of legacy data marts in existence; these data stores and the applications that rely upon them provide significant challenges to the data warehousing practitioner.
The typical legacy data mart grew up in the late eighties or early nineties, and was probably referred to as the executive information system (EIS). It may have been developed using a series of specialized tools for creating geographic or organizational pictures of the information of interest. The audience was usually limited to a few people in the organization maybe a particular department or a handful of executives. It was custom-built for one purpose and was usually quite difficult to modify. In exchange for a significant investment in customized programs and configuration, the end user got a system that provided a region-by-region breakdown of sales information, with drill-down capability and the ability to highlight problem areas based on data values.
What's wrong with that, you say? Nothing. These systems, however, are plagued by the same problems suffered by other legacy applications, including the following:
Technical obsolescence. Many of the proprietary tools used to build the EIS are no longer supported by the vendors who created them. As a result, the IS department has no viable capability to make alterations to the application to fit new requirements, and the application is no longer fully supporting the user. This problem is made worse by the fact that many of the EIS solutions that we developed a decade ago were one-offs, with each department in the organization choosing a different set of proprietary tools.
The solution for this problem is a systematic assessment of the platforms used for data mart delivery. Each application must be examined with a hard look toward the costs, benefits and usefulness of the application. Such an effort can be part of a larger project to develop an enterprise strategy for deploying a data warehouse. The technical architecture development techniques discussed in February's "Data Warehousing Horizons" column are useful here.
Documentation Atrophy. As time goes by, most applications move further and further away from the documentation that was originally written to describe their function. This is particularly problematic for the meta data definitions in data mart environments; as the user definition of a field in the EIS report or screen changes, the documentation is frequently not updated along with it. Just about everyone who regularly works with a sales reporting application has had to field many phone calls asking, "How did you calculate same-store sales for Q3 of last year?"
Frequently, this problem can be solved with a short-duration effort to "clean up" and republish the documentation. If this approach is taken, it is generally worthwhile to use Web technologies to publish the information and publicize it to the user community. Doing so serves a twofold purpose: it makes the documentation easier to use (and thus more likely to be used) and encourages feedback and refinement as a result.
A second approach is to reverse engineer the EIS into a more robust and structured meta data repository. This may be a largely manual process, but the long-term benefits may be worth the effort if the information becomes useful in a larger data warehouse context.
Redundancy and Inconsistency. These issues arise for any data mart that was developed outside a larger data warehouse context. Unfortunately, most data marts were developed with a limited scope and user community in mind, as departmental applications. And, of course, different user communities may have differing ideas about the definitions and calculations of what seems to be the same information in their data marts.
In such an instance, development of a common set of business questions and the source data to answer those questions (again, see February's column) can be a valuable exercise. Facilitating agreement among the user groups about the assumptions and definitions they use for their information is among the first steps toward a unified and rational approach to data warehousing. Eliminating redundant stores for each department can also relieve some of the administrative and maintenance burden for the IT organization.
The Challenge: Getting it Done
The high-level answer to the issues discussed here is summed up as taking a step backward in the life cycle of a data warehouse and identifying the business case, business questions and technical architecture that makes sense given the current state of the organization and the technologies it uses. How do you sell such a project to your management? First, try to demonstrate the cost savings that can be realized by eliminating the redundancies in your departmental data marts. Don't forget to include the additional user analyst time spent on reconciling one EIS against another and the IT support time spent digging through extraction and load programs to determine the disparities. Then you can move to the softer benefits of better decision making capability and quality of information. Good luck.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access