Data quality may not be as sexy as big data has been this past year. The technology is mature and reliable. The concept easy to understand. It is also one of the few areas in data management that has a recognized and adopted framework to measure success. (Author’s note: Check out Malcolm Chisholm's column on data quality dimensions.) However, maturity shouldn't create complacency. Data quality still matters, a lot.
Yet, judgment day is here and data quality is at a cross roads. Its maturity in both technology and practice is steeped in an old way of thinking about and managing data. Data quality technology is firmly seated in the world of data warehousing and ETL. While still a significant portion of an enterprise data management landscape, the adoption and use in business critical applications and processes of in-memory, Hadoop, data virtualization, streams, etc., mean that more and more data is bypassing the traditional platform.
The options to manage data quality are expanding, but not necessarily in a way that ensures that data can be trusted or complies with data policies. Where data quality tools have provided value is in the ability to have a workbench to centrally monitor, create and manage data quality processes and rules. They created sanity where ETL spaghetti created chaos and uncertainty. Today, this value proposition has diminished as data virtualization, Hadoop processes and data appliances create and prop up new data quality silos. To this, these data quality silos often do not have the monitoring and measurement to govern data. In the end, do we have data quality? Or, are we back where we started from?
To be viable long term, data quality tools need to expand and support data management beyond the data warehouse, ETL and point of capture cleansing. They need to embrace the new data management paradigm. Today and tomorrow's enterprise will place higher value on governance enablement and the ability to extend sophisticated and mature processing across the entire data management platform.
So while the rhetoric of late has been about ensuring the quality of data in the world of big data, the real test will be how data quality tools can do what they do best regardless of the data management landscape.
2013 looks like a defining year for enterprise data quality tools.
This blog originally appeared at Forrester Research.
I am with you on the firewall concept. It is the "all information" point that I will pose a question on the 'V' for volume dimension of big data that this implies. Should we institute the same policies and rules for all data regardless of it's variety, velocity and variability?
Is all data created equal? Current data quality processes may not be aligned and may require adaptation to account for new data sources and conditions. Also, new processes are most likely going to be needed for this very same reason. I say yes to a holistic and managed approach. However, we cannot be complacent in continually adapting the policies, rules, and processes.