A prospective buyer of data warehouse technology can learn a great deal from performing a benchmark specific to his or her firm's data center and operations. If an enterprise is contemplating spending millions of dollars with database warehousing and hardware vendors, it would be wise to perform a custom benchmark based on a firm's specific data profile. It provides an opportunity to verify that the investment will satisfy the requirements as well as an opportunity to negotiate with the vendor for concessions. However, if a firm already has significant experience with a vendor (for example, it already operates with the vendor's database or hardware), a benchmark is less useful or necessary, though by no means should it be ruled out on principle.

If a vendor thinks you are a serious buyer who may spend a million dollars, the vendor will often perform the benchmark at no extra cost (or minimal cost) to the prospect enterprise. Thus, the cost to a specific enterprise in terms of dollars changing hands in a transaction may be zero, provided the enterprise has built a credible case that it wishes to buy and provided the vendor succeeds in meeting the performance requirements.

Obviously, this is a delicate negotiation that requires planning and effort on the part of the buying enterprise – so there is a cost in time and effort to the prospective buyer; however, it is not a transaction cost in the market.

Along the same lines, in order to derive value from the benchmark, the prospect must be prepared to devote time and effort to defining the performance requirements, selecting and delivering the relevant data, and participating in the process. That is why a distinction is made between a transaction cost in the market (which is often zero) and a cost to the prospect in terms of staff time and effort (which may be significant).

To appreciate the effort involved and the kinds of questions to ask when performing a custom benchmark, visit the Transaction Processing Performance Council Web site (www.tpc.org). Look at the full disclosure report (FDR) for the public benchmarks at the 300GB, 1TB or 3TB volume points. What are your SQL queries? How much data is involved? What is your data model? Those who are interested in performing a custom benchmark must be ready to answer these questions. Before proceeding with negotiations to perform a benchmark, it is useful for the prospective client to have an inventory of its own data assets (data profile, growth prospects, applications) and an understanding of its performance requirements. If you are the prospect and have done your homework in these areas, then proceed.

Another thing to consider is that the benchmarking center of any given database or hardware vendor has the capacity to perform a couple of dozen benchmarks a year for individual prospective clients at the rate of one or two benchmarks every one or two weeks. Naturally, the larger vendors have more bandwidth, but it is not infinite. Therefore, the vendor must decide, largely based on input from the sales team, which two or three dozen major clients a year would make this effort worthwhile and result in a significant win (i.e., sale). That is what it looks like from the vendor's point of view.

Some vendors, such as DB2 (IBM) and Teradata (NCR), are highly competitive with one another across hardware, software or both. Including them on a short list would increase a firm's chances that the vendors would be willing to perform a benchmark at no incremental cost to the buying enterprise (other than the staff time and management effort). In another twist, if an end-user firm really wants to perform a benchmark, regardless of purchase decision, some vendors would be willing to rent excess capacity if it is available for that purpose. Finally, if the vendor believes you are benchmarking with its competitor, then it will be more likely to compete aggressively for the business.

As indicated, substantial public information is available at the TPC Web site under the TPC-H category. TPC-H represents the decision support benchmarks - "H" stands for "ad hoc" query.

In addition, the database or hardware vendor should be willing to provide the names and contact information for several clients who have visited its benchmark laboratory and allow you to talk to them confidentially. They will likely tell you that it is a worthwhile effort, even though it requires considerable effort on your part. A recurring theme – and data warehousing lesson learned – is that benchmarking is one of those tasks where you get out of it in proportion to what you put into it.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access