The Sad State of Data Quality

The topic of business data quality has been with us for more than two decades. Given the large number of vendor offerings dedicated to resolving data quality issues, one might be forgiven for believing that the problems have all but been resolved. A glance through the current literature reveals, however, that the problem of poor data quality is still very much alive.

Has the state of data quality in organizations improved over the past 20 years? Or is data management, and data quality in particular, still in a ghastly state with organizations and senior management feeling that the problem is overwhelming? Despite the clear concerns from business and a plethora of software vendors, there is surprisingly little concrete information available regarding the state of data quality in business.

It is important for both organizations and vendors to understand the current state of data quality in organizations, the degree to which they have implemented processes and software to improve it, the issues faced, the roadblocks to successful implementation and the business case for data quality. It is also beneficial to gain insight into the underlying reasons for the approaches and software tools selected as well as the available experience to date. In June 2009, The Information Difference conducted a survey aimed at gaining deeper insight into the views and plans of businesses regarding their current or planned data quality initiatives.

There were 193 respondents who completed the survey from all around the world, the majority from Europe (47 percent) and North America (44 percent). A high percentage (39 percent) of the respondents were from companies having annual revenues greater than $1 billion and represented a wide spectrum of industries. The survey revealed a number of findings worth exploring.

Data quality is believed to be good. One-third of respondents rate their data quality as poor at best and only 4 percent as excellent. Fully half (51 percent), however, considered their data quality as good, although this may be overly optimistic when set against other results from the survey. For example, one respondent told us, "Poor data quality and consistency has led to the orphaning of $32 million in stock just sitting in the warehouse that can't be sold since it's lost in the system."

Insight into the cost of poor data is lacking. A shocking revelation from the survey is that 63 percent have no idea what poor data quality may be costing them. Of those who attempted to calculate the cost consequences, 23 percent suggested the figure was below $1 million. This is surprising when compared with the anecdote of the orphaned stock in the warehouse. Just 15 percent thought it may be above $1 million.

Organizations are not measuring the quality of their data. Forty-two percent of organizations have made no effort to measure or monitor the quality of their data. Taken together with the lack of awareness of the costs of data errors arising from poor data quality, this clarifies the respondents' claim that "It's very difficult to build a business case."

One-third have an active data quality initiative. Surprisingly, 17 percent have no plans at all to start a data quality initiative, compared with 37 percent who currently have some form of data quality initiative in place. The remainder told us they plan to introduce data quality in the next one-to-three-year period.

Data quality has a broad focus. A remarkable 81 percent say that their data quality is focused wider than just "name and address," yet the latter is the area in which most vendors (more than 90 percent) currently have their base.

The plan for data quality spans the enterprise. Two-thirds of respondents plan for or currently have data quality spanning either the entire enterprise or one or more lines of business. One-third is focusing the initiative across the entire enterprise.

Two main barriers inhibit adoption. The top two barriers to adopting data quality were that "management does not see this as an imperative" and "it's very difficult to present a business case." This is interesting, given that the majority (63 percent) have not attempted to calculate the cost of data errors, and 42 percent make no effort to measure data quality.

The survey also revealed that less than a quarter of organizations (23 percent) have located data quality under the umbrella of data governance, which is probably where it should be.