In last months column, while highlighting the narrowing gap between business intelligence (BI) and customer relationship management (CRM), I discussed the fact that three of the CRM top 10 catchphrases (predictive analytics, customer valuation, personalization) happen to be BI capabilities. In the context of my discussion of the need for both operational CRM systems and data warehousing applications to bring these capabilities to fruition, I mentioned a white paper found on a popular CRM Web site that espouses the theory that world class analytics can be had without a data warehouse.
Naturally, this white paper caught my attention, and I took a closer look. The paper turned out to be authored by a vendor selling a type of access tool that incorporates a server-based middleware application with advertised robust query, reporting and dashboard capabilities. With these capabilities, a data warehouse that stores and optimizes (for reporting) a copy of the data from multiple operational sources is deemed no longer necessary. Nor does one have to invest in a costly data warehouse or central data mart, extract, transform and load (ETL) technology or expensive storage hardware. Whats more, adjustments to reporting information or the integration of new sources can be made without the complexity of costly data warehouse projects, giving almost immediate access to information. I realized that, although the conclusions drawn by the authors are at best misguided, the paper highlighted some common misconceptions about the data warehouse and about the requirements for sophisticated analytics - misconceptions that are likely to be present in many organizations new to data warehousing. Given that there are severe consequences to falling prey to these misconceptions, they merit refutation.
First, the view of the data warehouse as a central copy of data in operational systems, stored and optimized for reporting, is flawed. The conclusion that sophisticated middleware and access capabilities eliminate the need for the warehouse and/or data marts for robust analysis is incorrect. The data warehouse is actually the data store that provides us with a subject-oriented, integrated, time-variant, nonvolatile collection of data designed to facilitate an enterprise view for analytics. The data in the warehouse is not just a copy of operational systems information that has been centralized. It is integrated and cleansed; it is organized into subject areas and represents an enterprise view. While middleware is starting to play an increasingly important role in the data warehouse environment, middleware alone cannot handle the complexities associated with integrating very large volumes of complex data on the fly. If it could, there would be no data repositories needed or included in the leading master data management (MDM), customer data integration (CDI) and CRM products; nor would those organizations moving into operational BI bother with a data warehouse. Instead, all of these applications would follow the pull directly from the source and integrate on the fly virtual path suggested in the white paper and dreamed of by hopeful business users looking to cut costs and speed up delivery. Reality dictates that complex integrations, be they in the operational world or the analytical one, require some level of integrated data storage and additional tools to complement the middleware.
Also, the data warehouse is time-variant and nonvolatile, whereas the source systems typically house little history and the information changes constantly. The data warehouse provides the capability to incorporate historical data into analysis and to re-create exactly, at any point in time, past reports generated for business users. This capability is critical to credibility, even when source systems have changed, business rules have been altered and hierarchies reorganized. Without the historical snapshots preserved in the data warehouse and marts, trending, time-series analysis and past report re-creation are virtually impossible. Middleware cannot produce the sometimes decades of historical information stored in the data warehouse - none of which remains resident in the operational environment in an easily accessible way.
Lets look now to the suggestion that eliminating the data warehouse component when integrating a new source drastically reduces time to deployment and enables business users to see almost immediate returns on their information asset. While it does take time to design and build the data warehouse, it is commonly acknowledged that approximately 80 percent of the time in a warehouse project is spent in the data acquisition phase. Inherent in this phase are the activities of identifying the data source elements, profiling the data, identifying errors and determining (with the business) the best course of action for correcting them. Also included is the need to come to terms with consistent data definitions across business lines, develop and secure agreement on business rules for integration and determine frequency needs and history requirements. Without these time-consuming activities, you cannot achieve a single version of the truth emanating from integrated enterprise data. Nor can you really leverage your information assets to their fullest extent or achieve world class analytics. Without these time-consuming activities, you cannot guarantee the business immediate access to the new source, with or without a data warehouse.
In conclusion, beware of any vendor or consultant that promises scalable enterprise analytics while encouraging you to shortcut or skip altogether a comprehensive end-to-end architecture that includes both operational and analytic components as well as a mechanism for integrating the two. Sorry, but there is no silver bullet! While the delivery time may be shortened, the results will be as well: short on expectations, short on the enterprise view and short on quality and integration.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access