Benchmarks can be a source of marketing hype and spin unless they are carefully defined and audited by independent third parties. Even then, many hazards and risks threaten objectivity. In spite of these risks, industry- standard, independently audited benchmarks, such as those sponsored by the Transaction Processing Performance Council, have value. Used properly, benchmarks can drive development of new technology features that benefit real- world customers (not just "benchmark specials"), creating an economic reference point about cost, setting a performance bar at a point in time for a given software and revealing lessons about how to tune products that can be shared with the end-user community. Auditing of technology benchmarks is a subset of objective research and, as such, is the kind of activity likely to continue to be in the news. Business ethics – integrity in messaging, transparency in metrics and accountability in leadership – is arguably the latest post-Enron business and technology trend.

Ascential Software is launching an initiative to create an extract, transform and load (ETL) benchmark that would be objectively defined and audited. This is not a trivial undertaking; and, in many ways, the odds are against it. Such an undertaking faces a number of challenges, including the need for an objective audit, the issue of resisting the lure of "benchmarketing" and the inevitable task of making inferences from laboratory work to real-world experiences in client data centers. Nevertheless, because of the latest flap about integrity (an arguably objective need in the market) and because one of the vendors has stolen a march on the other by acquiring parallel technology, now is an opportune time to make sense of conflicting performance claims. The ultimate issue is whether the dynamics of unenlightened self-interest will be able to be contained by professionalism and objectivity.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access