Continue in 2 seconds

Reference Checking: The Gold Standard of Tool Selection

Published
  • July 01 2003, 1:00am EDT

Selecting data warehouse software is always a chore, and the process can often delay a project. This article will focus on the process of reference checking. Most organizations acknowledge that they will check references; but the reference-checking process is often disorganized and in many cases abandoned – sometimes because the organization already knows the vendor/product it wants to choose. Let's be clear, the right data warehouse software is a critical success factor for every data warehouse project. Without proper reference checking, the organization has not performed the due diligence required for such important decisions. There is an additional benefit to reference checking: You will pick up lessons learned and best practices that go beyond the best set of tools.

Alternatives to Reference Checking

Reference checking is by far the most cost effective, accurate and shortest route to evaluating products and vendors. Let's look at the alternatives. Note that none of these are mutually exclusive with reference checking, but they may not be necessary.

  1. Invite the vendors to present and demonstrate their products. The vendors will tell you all that is bright and beautiful about their products, never mentioning the trials and tribulations always associated with any significant software implementation.1 The demos will usually be impressive. You will be amazed by the speed with which the vendors are able to accomplish complex tasks (they have been practicing for months) and by the speed of response (they are only accessing 10,000 records). They will highlight all their ecstatic customer successes (never mentioning the failures or the customers with postal inclinations).
  2. Bring in the tools that are on the short list and exercise them in your own environment. This takes considerable effort and requires time for you to learn some tools you will never use. Additionally, some tools don't manifest their warts until they have been in productive use for some time. You will not discover these deficiencies and problems in the short time you have with the product. This approach will delay the choice of the tool and will probably delay the entire project.
  3. Let the vendors exercise the tools in your environment. Some vendors will accept the challenge, but then you have to decide how to rank the vendors who do not want to participate. Vendors may be afraid of failing, they may not have the personnel to enable them to participate, or they may not be hungry enough. If their participation in this bake-off is a prerequisite for being considered, you may be missing the best product.

Evaluating Vendors

Some organizations just look at the functions and capabilities of the tool and fail to consider the company that's selling it. However, you absolutely want to know about the vendors for the following reasons:

  • Support: Without good support, the tool will cause you trouble, and you may never realize its full capabilities. You will become frustrated and may experience significant downtime, terrible performance or the inability to make the tool work with your other software.
  • Stability: The vendor may be out of business next year, and you'll be left with a white elephant that will eventually have to be converted (an unpleasant nightmare).
  • Integrity: The vendor may be less than totally honest about the capabilities and ease of use of their products, comparison with the competition or future plans.

Selecting and Gathering References

Your first step in reference checking is to contact the vendors whose products you are evaluating to request a list of references. If they are smart and well organized, they will give you their best references – the ones who will say nice things about them and their products. Therefore, recognize that you will be getting a somewhat biased view and one that is not a representative sample of their customer set. That's okay. Even with this select group, by asking the right questions, you will be able to discover the dark side of the vendors and their products. Some vendors want to participate in the call; however, the discussion with the references will be much less candid with the vendors involved. Decline this arrangement.

If you do not tell the vendors the types of references you want, they are likely to give you some that are worthless. Tell them what you want and what you do not want. Please see the sidebar for a sample letter to a vendor to request customer references.

Example Letter to a Vendor to Request Customer References

Dear Vendor:

Congratulations, you are on our short list! Our next step is to talk with organizations actively using your product. We are looking for specific types of references. Please provide us with four references that have the following characteristics:

  • The reference need not be local (we will be using the phone, not visiting), and it does not have to be in our industry.
  • The reference should have been actively and productively using the product under consideration for at least six months. We don't want anyone in the throes of implementation, and we don't want anyone who is just testing/evaluating the product.
  • We are interested in a specific release; therefore, we want to talk with a reference that is using that release (indicate which release).
  • The reference must be on a similar platform (operating system and RDBMS) as we intend to use. Please do not provide an NT reference as we plan to be on UNIX (indicate which platform).
  • The reference must have at least as many users as we intend to have and a database of at least the size we plan (indicate number of users and/or size of the database). Note: This is to verify performance, not function or ease of use so you will want at least one of this type.
  • The reference must not have a financial or marketing association with your organization.
  • If possible, please provide references that also use the same products as the ones we have already selected (indicate the products you have already chosen).

Next, you want to find other customers who are using the products. If possible, connect with user groups, attend their meetings and talk to some of the more vocal members. Tell them honestly that you are evaluating products, and ask for their time to answer a few questions. You can interview them at that time or sometime after the meeting.

Finally, ask around and locate other companies using the products. It's especially useful to talk to installations with more than one of the vendor's products as well as those with competing products. These organizations are in an especially good position to tell you how the products compare.

The Process of Reference Checking

You don't have to travel to the reference site. Hosting a visit is, in many ways, an imposition on the reference; therefore, fewer references will be willing to give you information if your mode is to visit. It is far cheaper, faster and more effective to use the phone.

It's usually best to have one person make the calls. This provides a certain level of consistency to the questions asked, documentation and consistency in the conclusions.

Find out a little about the company you are calling including the industry, size and any information on their data warehouse applications that is publicly available. If the vendor gives you the name and number of a reference, the vendor has undoubtedly forewarned the reference of your impending call; however, don't make that assumption. Ask the vendor to alert the references that you will be calling. On your initial call to the references, you will introduce yourself and schedule a time for the actual reference-checking conversation. You will also give a very brief description of your environment and project, where you are in the selection process and the products you are considering. Thank them in advance for their time.

Don't ask the references if you can record the calls – the concern about being recorded will usually reduce the candor – and certainly do not record without their knowledge or approval. Use a headset or a speakerphone so you can have your hands free to take notes (don't even imagine you can remember all that was said). Allow time after the call to flesh out the notes.

Questions to Ask

Before you talk with the references, you must know what you will ask. It is important to be consistent in your questioning. Be sure to ask all the important questions. Don't waste the references' time, but do start off with the easy questions. The questions will vary based on the tool category. Don't ask questions if you already know the answers. Ask only questions that will make a difference in your decision. Be sure to thank the references for their time and ask if it would be all right to contact them again with any additional questions.

The box below contains sample questions you can use to build your own set of questions for the references (you will not be asking all of these questions). There are general questions to be asked for any category of tool, questions specific to the business intelligence (BI) tools, questions specific to RDBMSs and questions specific to ETL tools.

General Questions

  1. What hardware, operating system, operational RDBMS, network topology, etc. is used?
  2. If you are using multiples of above, do you experience any differences in performance or availability?
  3. What are your scheduled hours of availability (e.g., 24x7, 18x6)?
  4. Do you have service level agreements (SLAs)? What are they? Do you consider your system to be mission critical?
  5. How much downtime do you experience? Is unavailability the result of planned maintenance, unplanned maintenance, software problems or other problems?
  6. As a rule, do you continue to evaluate other tools?
  7. What other similar tools are installed and being used?
  8. How long have you had the tool in production?
  9. How long did it take you to roll out your application(s)?
  10. What was the composition of the team that installed the tool (including number and skill level)? Where do they spend their time? How long did it take for them to become proficient in the tool? Which positions did they hold before they were trained and how skilled were they?
  11. What is the composition of the team that supports the tool, number and skill level? Where do they spend their time?
  12. What other tools did you consider in this category? Why were these rejected?
  13. What were your criteria for selection?
  14. What was your opinion of the other tools?
  15. What kind of problems did you have with implementing the tool? How were those problems resolved?
  16. How easy was it to migrate to a new release? Did you ever have to go back to the previous release?
  17. What kind of problems do you have with ongoing maintenance?
  18. If you have multiple vendors, how well did the tools work together and how well did the vendors work with each other when there were problems? Was there finger pointing?
  19. Does the vendor maintain metrics on their software quality and do they share this information with you?
  20. Please comment on vendor support, vendor help desk, etc.
  21. Did you require consulting support for the product? Was the support from the vendor? What was the quality of their consulting staff?
  22. Do you have any general comments on the vendor that would be helpful in our selection process?
  23. What security capabilities do you use and how easy are they to administer?
  24. Do you measure your costs? How? What are the costs associated with this tool? Were the costs you experienced greater than what you were expecting or greater than what was budgeted?
  25. Would you describe your architecture? Is it two-tier or three-tier?
  26. If you started over again, what would you do differently?
  27. Do you know any other organizations that are using the product that might be willing to discuss their experience?

Questions Specific to Access and Analysis, Analytic and BI Tools

  1. What type of application do you have (e.g., report, operational, query, analytic, ad hoc, data mining)? How complex are the queries and reports? How much leeway do you give your users in writing their queries?
  2. What is your average response time?
  3. How do you measure response time (tools, utilities)?
  4. How many queries are run each day/week/month?
  5. What training was given to the users?
  6. Who provided the training?
  7. What is your assessment of the quality of the training?
  8. How do you evaluate the effectiveness of the training?
  9. How many users were trained on the tool?
  10. How many active users do you have? How many concurrent users?
  11. If you have significantly more trained than active users, what are the reasons for some of the trained users not actively using the tool?
  12. How would you characterize your users (power, casual, report reader, executive, etc.)?
  13. How does each type of user like the tool?
  14. Have you measured user satisfaction?
  15. How satisfied are the users? Any specifics?
  16. What do they like about the tool? What do they dislike?
  17. What do you like about the tool? What do you dislike?
  18. How many BI developers/support people are assigned to the project?
  19. Do you have canned queries and reports available to the users? Are these canned queries and reports being actively used?
  20. If you use both a Web implementation and a standard implementation, what differences have you seen (especially differences involving function)?
  21. On your help desk, what types of questions are asked?
  22. If you grew, what did it take to add a new set of users?

Questions Specific to RDBMS Tools

  1. How many tables do you have?
  2. Was your data modeled? How effective were the results of the modeling process?
  3. What do you schemas look like? Are they third-normal form, multidimensional, etc.? Why were these schemas chosen?
  4. How long does it take to perform specific operations (load/update/refresh times, response time for queries, number of concurrent users, workloads)?
  5. What is the size of your entire data warehouse? What is the largest table? (ask for both size in gigabytes and number of rows)
  6. If you are distributed, what are the capabilities of the RDBMS that assist in a distributed implementation?
  7. How well does this RDBMS work with your other software products (ETL and BI)? Are there any problems?
  8. Do you measure performance (resource use, response time)? If so, how do you do it?
  9. What is the ratio of disk to raw data?
  10. What is the ratio of summary tables to raw data?
  11. How much of your database is devoted to indexes? How much is devoted to summary data? How much to work space?
  12. Do you mirror your database?
  13. How many DBAs do you have supporting the data warehouse? What activities do they perform?
  14. How easy were hardware upgrades? How long did they take? Did everything go as planned? Were there any configuration issues related to the upgrades?
  15. If you grew, what did it take to add a new set of users and/or additional data?
  16. How often do you reorganize the data? How long does it take? Is the system unavailable while you reorganize?

Questions Specific to ETL tools

  1. What are your data sources?
  2. How long does it take to perform specific operations (load/update/refresh)? What is data rate?
  3. Do you have a batch window? If so, how long is it? Do you have any trouble working within that window?
  4. What is the size of your entire data warehouse? What is the largest table? (ask for both size in gigabytes and number of rows)
  5. How well does your ETL work with your RDBMS?
  6. Do you use the meta data capability of the ETL tool? If so, how do you use it?
  7. How extensive is your transformation process? How labor-intensive was the writing of the transformations?
  8. Are you using the data cleansing capabilities of the ETL tool? How well is it working?
  9. Are you using any additional data cleansing tools? If so, which ones, and how well are they working?
  10. How much data do you load/update/refresh? What is the load/update/refresh frequency?
  11. How many ETL developers do you have supporting the system? What are their primary roles?

Documenting Results

You will not have gone through the trouble of calling the references if the tools being considered do not have the basic functional capabilities you need. This means that the results of the reference calls will be the primary determinant of the selection – unless senior management has already made the decision. Using a standard template will enable you to easily compare the tools and vendors; it's too early to state conclusions. Gather the information, organize it and summarize it. Your report for the reference calls should include the following information:

  • Tool discussed
  • Interviewer
  • Date
  • Organization
  • Name
  • Title/role
  • Experience with tool
  • General impressions
  • Product strengths
  • Product weaknesses
  • Observations about the vendor
  • Ability to satisfy our mandatory requirements
  • Quality of vendor support
  • Quality of vendor training

In the evaluation process, every organization creates a set of mandatory requirements. If the tool or vendor does not satisfy a mandatory requirement, that vendor is dropped from consideration. Be careful not to specify a capability as mandatory if it is only desirable. Examples of mandatory requirements are:

  • Vendor that is financially stable and has a proven track record.
  • Good to excellent support.
  • Adequate performance.

Communicating Results to Stakeholders

The stakeholders include members of the evaluation team, the management involved with the project and anyone with an interest in the tool regardless of whether or not they will have any say in the final choice. You want to minimize the chance for assassins to dismiss or denigrate your choice as not being well thought out or well researched. Offer the stakeholders the ability to ask you questions. You could establish a temporary Web site with the findings, questions and answers.

Your report will have general observations, comments on how effectively the sites are using the tool, your observations and conclusions about the tool and the vendor. It will also include your recommendations for each tool being an acceptable choice. An appendix should contain all your notes from the calls.

By checking references in an organized fashion, you can shorten your time to select tools and vendors, which, in turn, will shorten your elapsed time to deliver the data warehouse. Checking references will also reduce the chances of choosing the wrong tool or the wrong vendor. Consider reference checking as the due diligence that should be applied to this critical set of decisions that can mean the difference between success and failure.

Reference:
1. This article could cause readers to infer some lack of integrity on the part of vendors. Note to vendors: these references do not apply to you but to your competitors.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access