Examining Dimensions of Data Quality: Completeness
The last article in this series looked at six authors’ definitions of two related dimensions of data quality (Timeliness and Accessibility). In this article, we’ll look at one of the most foundational dimensions, Completeness.
At a high level, Completeness is intuitive. The key to measuring Completeness (or anything in this world, for that matter) is to identify the data’s characteristics and then compare those known attributes at a later time to test whether they have changed, in this case whether they have changed from NULL to NOT NULL or vice versa.
The following illustration of a delimited file transmitted from one system to another shows the two primary types of completeness: row-level and column-level.
As you can see in the illustration, the file has four physical rows composed of three data rows and a header. Although you can calculate the number of rows as the literal four rows, we usually exclude the header from any counts or amounts to avoid confusion. If I included the header in the count/amount for each column in this file, how would I include ‘ITEM_PRICE’ in the sum of amounts ($15.25, $33.12, $24.95)?
When data is moved from one location to another, Completeness is a concern. In order to identify the loss of data, we need two measures of completeness for a two dimensional data set (e.g., table of data).
- Row-level Completeness: First and most importantly, to validate completeness we measure the count of physical rows in the file and then measure that again after we move the data to make sure we have all the observations.
- Column-level Completeness: Next, we count the values of each column or aggregate the values (if logically able), as seen in the ITEM_PRICE column in the previous illustration. This ensures that, on the whole, the data is the same.
Unlike the other dimensions covered so far in this series of articles, there is complete agreement with the primary concept of this dimension: column-level population.
In the financial services industry, row completeness is also very important (e.g., if a transaction was not recorded/moved, all associated revenues/premiums are likely understated). A consolidated list of concepts within the Completeness dimension must have the column and row levels, but in my experience, schema and table levels aren’t frequently measured.
- Redman, Tom. "Data Quality: The Field Guide," Digital Press 2001.
- English, Larry. "Information Quality Applied," Wiley Publishing, 2009.
- TDWI. "Data Quality Fundamentals," The Data Warehousing Institute, 2011.
- DAMA International. "The DAMA Guide to The Data Management Body of Knowledge" (DAMA-DMBOK Guide) Technics Publications, LLC, 2009.
- Loshin, David. "The Practitioner's Guide to Data Quality Improvement," Elsevier 2011.
- Yang W. Lee, Leo L. Pipino, James D. Funk, Richard Y. Wang. "Journey to Data Quality," MIT Press 2006.
- McGilvray, Danette. "Executing Data Quality Projects- Ten Steps to Quality Data and Trusted Information," Morgan Kaufmann, 2008.