Selecting data warehouse software is always a chore, and the process can often delay a project. This article will focus on the process of reference checking. Most organizations acknowledge that they will check references; but the reference-checking process is often disorganized and in many cases abandoned sometimes because the organization already knows the vendor/product it wants to choose. Let's be clear, the right data warehouse software is a critical success factor for every data warehouse project. Without proper reference checking, the organization has not performed the due diligence required for such important decisions. There is an additional benefit to reference checking: You will pick up lessons learned and best practices that go beyond the best set of tools.
Alternatives to Reference Checking
Reference checking is by far the most cost effective, accurate and shortest route to evaluating products and vendors. Let's look at the alternatives. Note that none of these are mutually exclusive with reference checking, but they may not be necessary.
- Invite the vendors to present and demonstrate their products. The vendors will tell you all that is bright and beautiful about their products, never mentioning the trials and tribulations always associated with any significant software implementation.1 The demos will usually be impressive. You will be amazed by the speed with which the vendors are able to accomplish complex tasks (they have been practicing for months) and by the speed of response (they are only accessing 10,000 records). They will highlight all their ecstatic customer successes (never mentioning the failures or the customers with postal inclinations).
- Bring in the tools that are on the short list and exercise them in your own environment. This takes considerable effort and requires time for you to learn some tools you will never use. Additionally, some tools don't manifest their warts until they have been in productive use for some time. You will not discover these deficiencies and problems in the short time you have with the product. This approach will delay the choice of the tool and will probably delay the entire project.
- Let the vendors exercise the tools in your environment. Some vendors will accept the challenge, but then you have to decide how to rank the vendors who do not want to participate. Vendors may be afraid of failing, they may not have the personnel to enable them to participate, or they may not be hungry enough. If their participation in this bake-off is a prerequisite for being considered, you may be missing the best product.
Some organizations just look at the functions and capabilities of the tool and fail to consider the company that's selling it. However, you absolutely want to know about the vendors for the following reasons:
- Support: Without good support, the tool will cause you trouble, and you may never realize its full capabilities. You will become frustrated and may experience significant downtime, terrible performance or the inability to make the tool work with your other software.
- Stability: The vendor may be out of business next year, and you'll be left with a white elephant that will eventually have to be converted (an unpleasant nightmare).
- Integrity: The vendor may be less than totally honest about the capabilities and ease of use of their products, comparison with the competition or future plans.
Selecting and Gathering References
Your first step in reference checking is to contact the vendors whose products you are evaluating to request a list of references. If they are smart and well organized, they will give you their best references the ones who will say nice things about them and their products. Therefore, recognize that you will be getting a somewhat biased view and one that is not a representative sample of their customer set. That's okay. Even with this select group, by asking the right questions, you will be able to discover the dark side of the vendors and their products. Some vendors want to participate in the call; however, the discussion with the references will be much less candid with the vendors involved. Decline this arrangement.
If you do not tell the vendors the types of references you want, they are likely to give you some that are worthless. Tell them what you want and what you do not want. Please see the sidebar for a sample letter to a vendor to request customer references.
Example Letter to a Vendor to Request Customer References
Congratulations, you are on our short list! Our next step is to talk with organizations actively using your product. We are looking for specific types of references. Please provide us with four references that have the following characteristics:
Next, you want to find other customers who are using the products. If possible, connect with user groups, attend their meetings and talk to some of the more vocal members. Tell them honestly that you are evaluating products, and ask for their time to answer a few questions. You can interview them at that time or sometime after the meeting.
Finally, ask around and locate other companies using the products. It's especially useful to talk to installations with more than one of the vendor's products as well as those with competing products. These organizations are in an especially good position to tell you how the products compare.
The Process of Reference Checking
You don't have to travel to the reference site. Hosting a visit is, in many ways, an imposition on the reference; therefore, fewer references will be willing to give you information if your mode is to visit. It is far cheaper, faster and more effective to use the phone.
It's usually best to have one person make the calls. This provides a certain level of consistency to the questions asked, documentation and consistency in the conclusions.
Find out a little about the company you are calling including the industry, size and any information on their data warehouse applications that is publicly available. If the vendor gives you the name and number of a reference, the vendor has undoubtedly forewarned the reference of your impending call; however, don't make that assumption. Ask the vendor to alert the references that you will be calling. On your initial call to the references, you will introduce yourself and schedule a time for the actual reference-checking conversation. You will also give a very brief description of your environment and project, where you are in the selection process and the products you are considering. Thank them in advance for their time.
Don't ask the references if you can record the calls the concern about being recorded will usually reduce the candor and certainly do not record without their knowledge or approval. Use a headset or a speakerphone so you can have your hands free to take notes (don't even imagine you can remember all that was said). Allow time after the call to flesh out the notes.
Questions to Ask
Before you talk with the references, you must know what you will ask. It is important to be consistent in your questioning. Be sure to ask all the important questions. Don't waste the references' time, but do start off with the easy questions. The questions will vary based on the tool category. Don't ask questions if you already know the answers. Ask only questions that will make a difference in your decision. Be sure to thank the references for their time and ask if it would be all right to contact them again with any additional questions.
The box below contains sample questions you can use to build your own set of questions for the references (you will not be asking all of these questions). There are general questions to be asked for any category of tool, questions specific to the business intelligence (BI) tools, questions specific to RDBMSs and questions specific to ETL tools.
Questions Specific to Access and Analysis, Analytic and BI Tools
Questions Specific to RDBMS Tools
Questions Specific to ETL tools
You will not have gone through the trouble of calling the references if the tools being considered do not have the basic functional capabilities you need. This means that the results of the reference calls will be the primary determinant of the selection unless senior management has already made the decision. Using a standard template will enable you to easily compare the tools and vendors; it's too early to state conclusions. Gather the information, organize it and summarize it. Your report for the reference calls should include the following information:
- Tool discussed
- Experience with tool
- General impressions
- Product strengths
- Product weaknesses
- Observations about the vendor
- Ability to satisfy our mandatory requirements
- Quality of vendor support
- Quality of vendor training
In the evaluation process, every organization creates a set of mandatory requirements. If the tool or vendor does not satisfy a mandatory requirement, that vendor is dropped from consideration. Be careful not to specify a capability as mandatory if it is only desirable. Examples of mandatory requirements are:
- Vendor that is financially stable and has a proven track record.
- Good to excellent support.
- Adequate performance.
Communicating Results to Stakeholders
The stakeholders include members of the evaluation team, the management involved with the project and anyone with an interest in the tool regardless of whether or not they will have any say in the final choice. You want to minimize the chance for assassins to dismiss or denigrate your choice as not being well thought out or well researched. Offer the stakeholders the ability to ask you questions. You could establish a temporary Web site with the findings, questions and answers.
Your report will have general observations, comments on how effectively the sites are using the tool, your observations and conclusions about the tool and the vendor. It will also include your recommendations for each tool being an acceptable choice. An appendix should contain all your notes from the calls.
By checking references in an organized fashion, you can shorten your time to select tools and vendors, which, in turn, will shorten your elapsed time to deliver the data warehouse. Checking references will also reduce the chances of choosing the wrong tool or the wrong vendor. Consider reference checking as the due diligence that should be applied to this critical set of decisions that can mean the difference between success and failure.
1. This article could cause readers to infer some lack of integrity on the part of vendors. Note to vendors: these references do not apply to you but to your competitors.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access