One of the most hallowed traditions of business intelligence (BI) projects is the vendor/tool bake-off. Vendors are gathered, demonstrations are conducted, endless meetings are held to discuss the merits and shortcomings of each candidate, more rounds of demonstrations are performed, tests are executed and finally two or three finalists emerge. These survivors are pitted against each other in a bake-off that often resembles televised wrestling in volume, ethics and head-banging action.

Information technology (IT) teams seem as addicted to these bake-offs as preteen boys are to their knee-dropping, neck- locking wrestling matches. One of our clients recently remarked that bake-offs feed IT teams' needs for analysis and empirical criteria. I agree, and I also think they feed a need for staying on the bleeding edge of technology and providing the ultimate protection against later recriminations regarding the selected tool. Not only that, some IT team members really enjoy the lavish and undying attention paid to them by the vendor sales force during the sales, demonstration and bake-off process.

All of these factors lead to an endless cycle of vendor evaluation, demonstration, bake-off and procurement. What doesn't get much attention during all of this marketing and sales frenzy is exactly how much team bandwidth is required to conduct one full product bake- off cycle.

Let's add some quick numbers. A typical tool selection process will yield between three and five potential contenders. We'll use four for our example. Each vendor will need at least one half-day session for the introductory product demonstration/sales pitch. This yields two days of first- round demonstrations. Just about the entire team will need to attend these demonstrations, especially representatives from networking, help desk/support and change management. It's pretty common to have ten people in the room for the demonstration. This yields twenty person-days spent in first-round demonstrations. Throw in the one to three people dedicated to researching the tools to yield the three to five candidates, including trips to trade shows and the meetings to evaluate the first-round demonstrations, and you are easily looking at forty person days just to get past the first round demonstrations.

Of course, by this time, the team is in the full product procurement "red mist." No one is thinking about all the other things that are not getting done. Instead, everyone is attending at least one two-hour meeting a day to discuss vendor A's latest press release or to hold a phone conference with vendor B's latest reference. Then come the second and third rounds of product technical demonstrations. When you get the to the bake-off stage, significant team resources must be dedicated to preparing the test parameters, platforms, location, network, data sets, data models, etc. Again, no one is really keeping track of how much time is being spent because everyone believes 100 percent in the requirement for, and the validity of, the full bake-off life cycle model.

The bottom line is that by the time you get a tool selected, you will invest 200 to 300 person-days in the effort.

It is common for clients to come to us with BI projects they want to accomplish in six months with a team of five or less and the requirement to bake-off an ETL tool and an access and analysis tool in the process. This speaks to me to the level of naïveté existent in the market regarding the number of person-days required to do a full life cycle bake-off and to the level of blind faith people have in the process.

The next time you are faced with picking a tool:

  • Be realistic about the amount of time required to accomplish a full life cycle bake-off. It is not going to happen in two weeks and, in some cases, not even in two months. If you don't have 200 to 300 person- days in your project plan and resource allocation dedicated to a full life cycle bake-off, you are setting yourself up for missing your deadlines.
  • Consider short-circuiting the process. One of my peers who conducts these product evaluations and bake-offs for a living has stated that in one major product category, one vendor's product almost always wins. What is the point of investing 200 to 300 person days repeating the same result? If the only reason you are doing a full-scale bake-off is to provide protection and rationalization for your choice, you should meet with your steering committee to discuss alternative ways to accomplish this goal.

Consider forming a two-to three-person strike team to research and select a tool in a high-speed, direct manner. It is much less expensive to bring in an unbiased tool evaluation specialist consultant for a few days to make a quick, direct selection than it is to tie up your entire team for weeks.
If you select a direct route to a selection, you should always include a "proof of performance" clause in the purchase order or conduct a proof of concept (POC) prior to signing the purchase order.

The direct route may not offer all the bright lights, thrills and high drama of the cage match product bake-off, but it will save you critical person-days of effort that can probably be better expended elsewhere.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access