This is my first column for DM Review, and I am delighted to have the opportunity to share my experiences working as a data warehouse practitioner; I hope that you will share yours with me as well. Data Warehouses That Work will hopefully transcend many of the fragmented ideas and disciplines in data warehousing today and get right to the heart of it designing and building analytic architectures that don't just meet specifications, but delight and even amaze the people who use them. We'll explore how to deliver technology that can be used as leverage to bring information to an organization in ways that truly make a difference.
The whole point of a data warehouse is to provide an environment where people can analyze information for decision making. This definition is expanding rapidly to include much more integrated analytical processing, closed-loop systems and even standalone analytical robots using rules engines. But there is a problem. The end-user part of the equation is often an afterthought. Too many data warehouse initiatives stop with the data disciplines modeling, gathering, cleansing and storing data. That's only part of the solution. Data warehouses that work must be in some way transformative, perhaps even capable of changing the nature of work at a fundamental level in an organization. Granted, we never have the resources or the time to do anything quite so magnificent. That's the impasse at which the industry is stuck. What we can do is find areas where we can work with groups that are committed and together find ways to implement truly unique and powerful applications driven by the data warehouse.
In my experience, the largest hurdle to overcome when developing a successful data warehouse is the information technology/database perspective. Many years ago, the applications that are now supported by data warehouses were developed and paid for by businesspeople in marketing and finance, not information technology (IT). The applications themselves, not the database or the process, took center stage. When data warehousing moved into the mainstream 10 years ago, development moved to IT; and the database, ETL, meta data, servers, storage, security and, especially, the vendors of those products moved to the forefront. Though each of these is critical to your success, the combined gravitational field they form diverts attention from the softer side of the effort getting it right.
I can recall many data warehouse diagrams, architectures, methodologies and all sorts of depictions starting on the left side of the page and using the entire space to illustrate the technical process. In these diagrams, there are source systems, ETL, staging areas, meta data, data marts, ODSs, networks, hierarchical storage and "presentation" layers, at the very least. I can dimly see a few poor users in the lower-right corner, at the far end of a blizzard of right-pointing arrows, staring at a PC. That, unfortunately, is a very accurate depiction of the depth of understanding these implementations display for the work that people perform. Sadly, these diagrams are still common. As an industry, data warehousing has trivialized the work that people do and not served them very well.
Let's picture this diagram reversed, with the emphasis now on how information is used. The relative size of the back office shrinks, the arrows point back and forth, and the harried user with PC is replaced by a swarm of downstream analytical applications, with connectors to operational systems, Web sites and external parties. This is a more accurate picture of what organizations really need from their data warehouse initiatives. When actually seeing a data warehouse that works, one wonders how these opportunities could have been overlooked.
Perhaps the name "data warehouse" is at fault, applying the metaphor of a physical structure to describe the sorting and storing of information. When the term was first coined in the '80s, real warehouses were structures designed for storing massive amounts of inventory. Today, inventory doesn't just sit, warehouses support supply chain optimization, continuous replenishment and just-in-time requirements. In other words, warehouses in the supply chain have been transformed from largely passive repositories of goods to active, dynamic connectors in an ever-increasing real-time flow of products. The same transformation can happen in data warehousing, from passive collection and storage to one of dynamic flow and optimization.
There have been astonishing advances in data warehousing in the past 15 years. The size, performance and reliability of the technology are mind-boggling. In a way, it's become a victim of its own success. The more the state of the art advances, the more isolated the data warehouse becomes in the realm of technology and technologists. The people and processes that use information for analytical purposes are still waiting for these highly evolved repositories to help them. The future of data warehousing is on the right side of the page. The user part of the equation is where the opportunity to employ and integrate a whole new set of fascinating technologies to get those data warehouses working is solved.
As time goes by, I'll elaborate on real-life examples from my practice and from my colleagues' practices to illustrate how to apply this concept to your own projects. I will focus the need for end-to-end solutions in business intelligence (BI), closed-loop systems and analytical applications. I'll explore how successful organizations have managed to break through this pattern. In some cases, leading-edge techniques and tools were applied, such as real-time alerting, dashboards, graphical modeling tools for nontechnical people, sensing robots, event managers and even specialized Web crawlers. In others, organizations applied the basics but kept the focus squarely where it belonged from the start on achieving real value. Next month, I'll describe how a client completely changed they way they did business using their data warehouse in 1996.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access