November 7, 1940, at approximately 11 a.m., the Tacoma Narrows suspension bridge slowly began to sway. Within seconds, the swaying turned to twisting and soon after, the twisting to violent rotation. After just three months of use, the third largest suspension bridge in the world crashed into Puget Sound. Engineers later determined the cause of the disaster to be a stiff 30 mph wind, a common occurrence for this area. A detailed model of the bridge had been constructed, tuned and constantly revisited early in the project a perfect scaled-down prototype of the real thing. So what went wrong? The engineers made the mistake of never placing the prototype in a wind tunnel to test its aerodynamic stability. A perfect prototype was subjected to imperfect testing. The same could happen with a data warehouse, for it has the potential to fall just as fast and crash just as hard.
If not built with the correct concepts in mind, a prototype is nothing more than a confidence-building, dog- and-pony prop. Prototypes, in any discipline, should be carefully and practically designed with clearly defined objectives. There are several reasons you might build a prototype. You might want to demonstrate to management the possibility for return on investment, prove a complex data model or even identify performance opportunities. Just because you're focusing on one objective in particular doesn't mean you can't benefit from all.
Whatever the primary objective for building a prototype may be, one theme should prevail. Put some serious thought into what can be learned from the exercise in order to increase the chance for success. Don't just go through the motions. The following suggestions should help in thinking through and implementing this crucial, but often neglected, stage in warehouse development.
Build it Early
Build your prototype immediately after or in the later phases of requirements gathering. Remember, the goal is to learn something valuable from the experience. The earlier you shine a flashlight on those data monsters lurking in the closet the better. Although this may mean going back and redoing some requirements gathering, in the end you'll have a more accurate and useful system. In many warehousing efforts, months go by before a single byte of data lands on a disk. Project teams spend enormous amounts of time gathering requirements, evaluating software, analyzing risk the list goes on and on. All of these activities, while important, will become much more valuable when written from the real- world perspective of a practical prototype.
Tool selection is another factor that impedes the progress of an early prototype. If an ETL tool or even RDBMS selection seems months away, proceed without it. Build the prototype in Microsoft Access if necessary; just get it done quickly.
Use a Lean Team
Speed is of the essence. You don't want to be bogged down with the old "too many chefs in the kitchen" scenario. Designate a small team of your best and brightest, making sure to mix technical gurus with competent business analysts. For most situations, three or four individuals are adequate. This lean, mean, breakthrough team will create a prototype that spearheads all of your upcoming efforts. Some warehouses are born through the efforts of a single industrious individual on his/her desktop database. A small, talented team with decent synergy can produce remarkable results.
The last thing your breakthrough team should worry about is system permissions, standards or otherwise conforming to a production or even pseudo-production environment. If possible, obtain hardware that is completely at the team's disposal. This means being able to recarve, reconfigure and reboot whenever they see fit. If this type of control is not possible, ensure that those individuals who do have jurisdiction are an integral part of the team. If this presents a problem, bring out the red flag because the success of the overall project is now in jeopardy. Prototypes can reveal a great deal about the political struggles that lie ahead.
Focus on Key Dimensions
Whether it is a data mart or enterprise warehouse, the model for your prototype does not need to include every targeted data element. Doing so would cross the line from prototyping into full-scale system development and impede progress. Instead, focus on those data elements that best represent the overall spirit of your project. For example, one would expect the prototype for a warehouse that's supposed to improve customer relationship management to focus on the customer dimension. Likewise, it is not important for the dimensions you choose to be populated with all attributes, even though they may already be included in the model. By carefully pruning the targeted dimensions and attributes, you will save valuable time in the design and loading of your prototype.
Start with Realistic Source Data
It's always good to take care of data integration issues before focusing on load or extract performance. Tuning for performance on a faulty data model will mean doing it all over again when the model is corrected. As all developers know, it is easier to work with small quantities of data. Instead of loading all 10 million customers, start with about 200,000. Be sure to use actual source records, doing your best to achieve an accurate sampling across the data set. Some prototypes are built with fabricated data. This may work well as a show-and- tell piece for upper management, but it will not teach you much about the challenges that lie ahead. You may find yourself demonstrating a masterpiece that is next to impossible to produce. Once a small amount of data is proven, don't be afraid to turn your prototype into more of a proof of concept, testing performance through increased volume.
Apply a Front End
Demonstrating your prototype to key individuals is not a bad thing. First, it can make team members and end users aware of general project direction, creating a consistent vision of project goals. Second, it demonstrates project potential, oftentimes an important result when funding is on a "temp-to-perm" basis. For the most part, no one will be interested in looking at raw tabular data direct from your warehouse tables. It will be much more impressive to apply a slick front end, particularly if your prototype involves a mart. With today's OLAP technology and the right individuals behind the wheel, this may require a lot less time than you might think. Vendors are usually more than willing to offer a trial period when licensing terms have not yet been settled. Some will even provide a few days of free consulting to ensure the trial is a success.
Revise and Move Forward
Your prototype is complete, and both end users and upper management are ecstatic. The budget has been approved for major, full-scale development. What now? Before you begin barking out orders for full steam ahead, take a step back and evaluate what was learned. Sit down with your team for a good old-fashioned brainstorming session. How did the data model hold up? What was learned about overall data cleanliness? What performance problems were discovered? These and many other questions should be considered in order to maximize the usefulness of the prototyping effort. In general, what did you learn that you didn't know before?
What went wrong with the original Tacoma Narrows suspension bridge project? Simplification aside, the prototype did not prove itself to be useful. It failed to uncover information that was clearly available to those who sought it. Don't let your project become another collapsing bridge. Utilize every opportunity to learn as much as possible, as quickly as possible, about the task at hand, allowing your prototype to show you where the vulnerabilities lie. Strengthening these areas will prove your prototype to be one of the most useful tasks of the project.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access