You don't have to travel to the reference site. It is far cheaper, faster and more effective (because you can talk to more references) to use the phone. It's usually best to have the same person make all the calls. This provides a certain level of consistency to the questions asked and to the documentation in the conclusions.

Find out a little bit about the company you are calling, including the industry, size and any information that is publicly available on what they are doing. If the vendor gives you the name and number of a reference, the vendor has probably forewarned the reference of your impending call; however, don't make that assumption. Be sure to ask the vendor to alert the references that you will be calling. On your initial call to the references, you will introduce yourself, schedule a time for the conversation if your initial call is inconvenient for the reference, give a very brief description of your environment and project, where you are in the selection process and the products you are considering. Thank them in advance for their time.

Don't ask them if you can record the call - the concern about being recorded will usually reduce candor. Certainly do not record without their knowledge or approval. Use a headset or a speakerphone so you can have your hands free to take notes. Be sure to allow time after the call to complete and correct your notes.

Questions to Ask

Before you talk with the references, you must know what you will ask. It is important to be consistent in your questioning. The following is a set of sample questions you can use to build your own set of questions (you will not ask all these questions).

  • What hardware, operating system or network topology do you use?
  • Describe your architecture and infrastructure.
  • If you are using a variety of architectures and platforms, do you experience any differences in performance, availability or administrative activities? Would you elaborate on these differences?
  • What functions or types of applications are you performing with this product?
  • What are your scheduled hours of availability (24x7, 18x6)?
  • Do you have service level agreements (SLAs), e.g., availability, performance? What are they? Are they enforced, and if so, what has been the experience with the SLA?
  • Would you consider your system to be mission critical? Why is the system mission critical, or why not?
  • How much downtime do you experience? Is unavailability the result of planned maintenance, unplanned maintenance, software problems or other problems?
  • As a rule, do you continue to evaluate other software in this category? Why?
  • How long have you had the product in production?
  • How long did it take for you to install the software and make it usable?
  • What was the composition of the team that installed the software regarding number and skill level? Where did they spend their time? How long did it take them to become proficient? What positions did they hold before they were trained, and how skilled were they? Did the team members use any other product in this category before this one?
  • What is the composition of the team that supports the product? Where do they spend their time?
  • What other software did you consider? Why were those rejected?
  • What were your criteria for selection?
  • What was your opinion of the other products?
  • What kind of problems did you have with installing the product? How were those problems resolved? Did the vendor come on site? Did they charge extra for their assistance?
  • How easy was it to migrate to a new release? Did you ever have to go back to the previous release? If yes, why?
  • What problems do you have with ongoing maintenance?
  • If you have multiple vendors, how well did the products work together, and how well did the vendors work with each other when there were problems? Was there finger-pointing?
  • Does the vendor maintain metrics on their software quality, and do they share this information with you?
  • What has been your experience in receiving information on quality and other issues from the vendor?
  • Did you require consulting or contracting support for the product? Was the support from the vendor or from a third party? What was the quality of their consulting staff?
  • Do you have any general comments on the vendor that would be helpful in our selection process?
  • What security capabilities do you use and how easy are they to administer?
  • Do you measure your costs? How? What are the costs associated with this product? Were the costs you experienced greater than what you had expected or greater than what was budgeted?
  • If you had to start over, what would you do differently?
  • Do you know any other organizations that are using this product that might be willing to discuss their experience?
  • Do you get any benefit out of being a reference? If so, what does the vendor give you?  

Be sure to thank them for their time and ask if it would be all right to contact them again with any additional questions.  

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access