When building data warehouses, most of us are driven to strive for perfection. We want to make sure everything in the warehouse is exactly right. We want to define every user requirement down to the smallest detail, analyze each data source in depth and choose the exact hardware and software platforms. If everything isn't exactly right, our users will be frustrated. But, the closer we can get to a perfect initial design that meets all our users' needs, when we finally roll out the results we will surely reach the ultimate goal of application development: very satisfied users. Right? Wrong. While the goal is, of course, very satisfied users, striving for perfection is not the right approach to achieve that goal. That approach leads to an endless sea of work, most of which turns out to be misguided or even wasted. We have endless meetings about every last piece of data that might interest us. We have endless design reviews. We even have endless discussions about what the screens should look like for end-user access. The reason much of this effort is misguided is because it's fundamentally wrong to focus on having a perfect data warehouse when you first roll it out to users.

Striving For Perfection
An Imperfect Goal

There are two main reasons why striving for perfection is the wrong approach. First, it's really not possible to build a perfect warehouse because end users often don't know exactly what they want (a fact that IT staff always seems to complain about). So, how can we be expected to perfectly match unknown requirements? However, don't blame the users. From a general business perspective, users want the information needed to make decisions. But users can't possibly foresee every decision they will need to make. If we're going to be successful building solutions, we'd better understand that users can't know all their detailed DSS requirements, and we need to stop complaining about it.

Second, even if it were possible to have your warehouse perfect when you first roll it out, it wouldn't be perfect for very long. Assume you were able to wave a magic wand which enabled you to build something that suited end-user needs perfectly. Can you pat yourself on the back, claim you're done and move on to the next great challenge? No. Within a period of about six months, two things will have happened. First, the users will have become more familiar with the warehouse, which will lead them to want more from the system than they originally wanted. Additionally, many users will initially work with summarized data. However, they will reach a point where they've gleaned most of the value from the summarized data and want to dig deeper and ask more detailed questions. These are questions they hadn't thought of before, and the questions might not be easily answerable with the current warehouse configuration. The warehouse would no longer be perfect.

In addition to the users becoming more savvy, the second thing that will have happened is that the business environment will have changed. Different types of knowledge will be needed. Over time, what might have been critical information can become obsolete, and what wasn't even on the horizon during the requirements gathering may suddenly become critical. As an example, telcos now want to analyze not just total minutes of usage per month per customer, but also want ways to determine which of those minutes were voice calls and which were calls to an ISP. In the retail industry, retailers now want to analyze how their e-commerce initiatives are impacting sales in their traditional stores. As the rate of change in business continues to accelerate, the demands we place on our warehouses and marts will also continue to change. So, even if something was perfect once, it will not be perfect for very long.

The Culprit ­
Static Applications

The motivation behind trying to make the warehouse perfect for the initial roll out comes from the flawed notion that if we don't get it right initially, it will never be right. This comes from a traditional way of thinking about applications. Aside from the occasional modification to an application, we are accustomed to viewing applications as largely static. Once built, they don't really change much. We're accustomed to thinking that we're going to be stuck with what we initially get until someone decides to throw out the system and replace it with a brand new environment.

We must rid ourselves of such outdated modes of thought and embrace a different way of thinking about and building warehouses. The real solution is to understand, particularly in the decision support arena, that user requirements are highly dynamic, and the solution must be designed to be highly dynamic. The term I use to represent such flexible and scalable solutions (and which is the polar opposite of a static solution) is an "organic" solution ­ a solution that takes on a living, breathing characteristic, just like the organizations they are designed to serve.

An Organic Approach

To be successful, we have to do less. Yes, that's right ­ less. We shouldn't try to get every detail nailed down initially. It's not really possible to do so, and it's not the right goal anyway. The correct goal involves the following:

  1. Understand the basic requirements.
  2. Design a highly scalable solution.
  3. Deliver the first piece of that solution into the users' hands quickly.

This first piece won't be perfect or fully complete, and it's not intended to be. It's just intended to give users incrementally more value than they had previously. Then, by actually using the warehouse, users can give you real-world feedback on what is truly valuable and what isn't.
That's just the first step. To reach the ultimate goal of having very satisfied users, and to take into account changing business requirements, you must interactively deliver the rest of the solution while letting the warehouse organically grow and evolve just as your needs evolve. A key to building applications organically is that you must philosophically treat each iteration as if it were the first ­ that is, the goal is not perfection, but rather to add incremental value. Remember, even in biology, evolution is not a progression toward perfection. It's simply a process to help ensure that entities are best suited to thrive in their current environments.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access