The February issue of DM Review celebrated "15 Years of Excellence" with a description of developments and challenges in data warehousing. This column continues with a brief review of information quality (IQ) in data warehousing and trends that are increasing business intelligence (BI) effectiveness through proactive IQ.
Management's Need for Quality BI
Management has pent-up needs to understand how well the enterprise is operating. Operational systems support operations processes but cannot directly support the analysis of the operations. BI must deliver information in ways that support the strategic and tactical processes of understanding customer or product and service trends, including fact-based management decisions.
In the 1970s, organizations created "management information" functions to deliver information. In the 1980s, "information centers" sprang up, the precursor to modern BI environments. Executive information systems (EIS) came into being to extract information and present key business indicators to executives, leading to today's dashboards and data warehouses.
BI and IQ
Early data warehouses exposed major IQ issues:
- Excessively redundant and disparately defined operational databases consumed huge effort and budget in extracting, transforming and loading the data warehouses.
- Missing, invalid and inaccurate data caused huge data cleanup activities but showed that you cannot correct it all.
- Duplicate customer records, both within and across multiple databases, caused additional high costs in matching, reconciliation and synchronization of data.
One IQ lesson learned was that the department's IQ does not equate to enterprise quality information. "My" data may be okay for the department but could cause "your" processes to fail. Integration efforts simply could not resolve all the issues. These data warehousing experiences proved the common definition of IQ as "fitness for purpose" was not sufficient for information quality. Data warehousing shows that for information to be high quality, it must meet all information consumer needs, not just the "information producers."
Organizations with these problems are not managing information as a strategic resource. Neither are they applying sound quality management principles to information as a shared product.
The Value Proposition for Quality Information
The driver behind BI is a simple fact. The value of any resource is derived only when we apply the resource or put that resource to work. Money has value only when we spend or invest it wisely; people contribute value when they perform their work; a manufacturing plant has value when it is producing products. Information has value only when knowledge workers or applications retrieve it and apply it to perform work correctly or make intelligent decisions. But there's a catch. Information without quality is dangerous. It can mislead thoughtful knowledge workers to make wrong decisions.
The Evolution of IQ Practices in BI
Most early data warehouse quality approaches were reactionary, correcting data in the data warehouse or in the staging area before loading. This early and immature data quality approach parallels early quality practices in industrial age manufacturing. Quality meant putting inspectors at the end of the assembly line to inspect and pull defects off to be reworked or scrapped if they could not be fixed. Hence, we have the "inspect and correct" or "scrap and rework" approach.
Manufacturing quality management matured, replacing the inspect-and-correct approach with a proactive "design quality in" approach. This eliminated the costs of inspection and scrap and rework.
Leading-edge organizations are proving that designing quality into processes and operational systems at the source is a more cost-effective way to solve IQ problems. Some lessons learned:
- Use the operational data store concept to create an enterprise-strength operational database so you can reengineer and replace obsolete, disparately defined data structures in a gradual, phased-in approach. This reduces the costs and risks of integration while eliminating a major cause of poor quality information.
- Solve IQ problems in the source processes - not downstream. This prevents inconsistency problems of data in source and target databases.
- Conduct data cleanup as a one-time project for data in a given data set.
- Improve the source process and application with error-proofing techniques to prevent defects.
- Provide training to information producers' managers, helping them understand their downstream information consumers' IQ requirements and equipping them with methods and techniques to improve their processes.
- Implement accountability for IQ in every manager's job description. Provide them with training and give them six months to a year to put IQ principles, process improvement and staff training in place. Then hold them accountable.
Organizations today are doing one of two things: 1) building more data warehouses without proactively addressing IQ problems or 2) solving IQ issues at the source, enabling their organization to become intelligent learning organizations.
What do you think? Let me hear at Larry.English@infoimpact.com.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access