Selecting the right business intelligence tools for your users is challenging. There are dozens of tools on the market today and too little time to research and evaluate all of them. What makes the task harder is that vendors don't readily convey their products' true strengths and shortcomings. Most vendor representatives are good people. In general, they want to do the right thing. But they also have a job to do. In most cases, vendors don't lie about their products; they just don't tell the whole truth.

So how can you extract the truth from vendors? How can you be sure a vendor's product is appropriate for your application needs? How can you discover that fatal flaw that will cause the product to fail once deployed in your environment?

Listed below are seven techniques that can help you gain an advantage in the vendor mating dance. Some are new twists to time-tested strategies, while others are lesser-known tricks of the trade.

1. Ask for Futures. Since vendors won't tell you product shortcomings directly, you have to be clever and get them to tell you this information indirectly. One way to do this is to ask the vendor what major enhancements they are planning to deliver in their next release or two. Sign a confidentiality agreement if necessary.

The future enhancements will point to the major areas of pain suffered by their current customers. These enhancements will give you a sense of the vendor's relationships with current customers as well as the maturity of their product. If the enhancements are significant, such as an architectural rewrite to support improved scalability and performance, then the product may be less than satisfactory or stable now or the immediate future.

2. Talk to Developers. The people who will provide the most straightforward information about a product are the techies who developed it. These folks have a difficult time being anything less than candid about a product, unless they are on a sales commission as well. The techies will give you the whole product picture--with warts and everything--not just the silver lining. That's why the marketing folks typically keep developers under lock and key in some remote, off-site development lab.

But you should insist on speaking with the product developer or manager. If you encounter resistance, propose having your most technical developers to speak to these folks. This is slightly less threatening to the vendor and can still generate lots of good information.

3. Talk to Competitors. Another good place to ascertain a vendor's vulnerabilities is from their closest competitors. Vendors spend a huge amount of time competing against two or three other firms for identical customer accounts. Through customer benchmarks and conversations with joint prospects, they get to know each other's product strengths and weaknesses quite well. In addition, most vendors have successfully recruited sales, marketing and technical people from their direct competitors. These folks provide critical intelligence about their former employers, their product lines and strategies.

Ask your prospective vendor who their chief competitors are (and press them if they say they don't compete against anyone!). Then, call up these competitors and ask for a product briefing. During the discussion, make sure you ask the vendor to comment on the primary weaknesses of their chief rivals. (Don't bother asking about strengths because they are incapable of discussing the advantages of their competitors!)

You'll get a biased perspective for sure, but you'll also get some ammunition you can use in conversations with your prospective vendor. You'll also discover new products in the marketplace that might better serve your needs or trigger ideas about how to restructure your project and requirements.

4. Conduct In-House Prototypes. It goes without saying that every product looks good in a demo. Therefore, the only way you can truly evaluate a product is to build a prototype.

But take caution here. Make sure the vendor comes to your site and builds the prototype with your data under your close supervision. You need to make sure they are building real functionality. Assign one of your own staff to observe what the vendor does when building the prototype, both for training purposes and surveillance. Also, give the vendor a specific time frame and an objective, such as to replicate 10 of your most complex reports.

5. Beware of Checklists. Prospective buyers often try to extract the truth from vendors by creating detailed checklists of product functionality and services. Unfortunately, checklists typically backfire because they are the perfect vehicle to allow vendors to stretch the truth with impunity no matter how exacting your questions.

Vendors will always check "Yes" to a question if there is the remotest possibility that they can deliver the requested functionality or feature. Checklists often don't expose the fact that a vendor can only deliver specified functionality through extensive coding or integration with third- party products. Checklists that are granular enough to cover these issues often become too large and unwieldy to be of much use.

A better approach (and one that requires less up-front work on your part) is to create a "Request for Information" that contains open-ended questions that ask the vendor to address your strategic issues. Vendor responses to open-ended questions will vary widely, giving you a better sense of the vendor's strategy, approach and interest in capturing your business. You should also allow vendors to present their strategy in person, since some vendors may not be able to adequately convey their ideas in writing and vice versa.

6. Seek Outside Evaluations Carefully. Most analyst and consulting firms perform product evaluations, which can provide more insight into vendor products than the vendors are willing to share.

But use caution, here. Did the vendor pay the analyst or consultant to review the product in the form of a "white paper?" These white papers are a very common vendor marketing tactic. Since the vendor has the right to review the outline and drafts of the paper, the vendor can exert considerable influence on what is said (or not said) in the paper. Even the most reputable analysts or consultants will tone down or skirt any vendor shortcomings in such white papers.

Even if the review appears in an objective trade journal, it would be wise to check what work the analyst has done for the vendor in the past six months or whether he or she is on retainer with the vendor. Although most vendors legitimately use retainers to get outside help with product positioning and strategy, some also do it to try and buy favor with key analysts. Objectivity is a very subjective thing!

7. Extend Vendor References. A standard procedure is to check all vendor references. But remember, vendors carefully preselect references that are likely to say only good things about the product (although sometimes vendor references backfire on the vendor!). Consequently, you may not always get all the information you need to make the most informed decision possible.

So, you need to go a step further and contact customers who aren't on the vendor's reference list. These may be companies that are having difficulty deploying the vendor's product, have decided not to purchase the product after an extensive evaluation or never comment publicly about products as a matter of policy. So how do you get these names?

One way is to check the vendor's Web site for old press releases, white papers, reports, etc., to see if you can find customer or company names that aren't on the current reference list. Another method is to search trade papers, such as Computerworld and DM Review, that run a lot of case studies. Once you identify an individual and company that are associated with the product in some way, then you can call the company switchboard to get connected with the individual. If the person has left the company, a departmental secretary is often willing to connect you to their replacement.

Another place to look is with the vendor's key technology partners. Often, vendor partners will share the same customers. (This is how they discovered they should partner in the first place!) Ask the partner for joint references with the vendor whose product you are evaluating.

Finally, there are some companies that seem to purchase one of almost any kind of product. These are large companies with huge information technology (IT) departments that are charged with using IT for competitive advantage. Examples are AT&T, Fidelity and U.S. West. If you have a friend or acquaintance in these companies, you might be able to find a person who has experimented with the product you are considering.

In summary, don't let vendors dictate the rules of the game. Use the techniques above to extract the truth from the vendor. These techniques can help ensure you get the information you need to make informed product selections.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access