Businesses need quality data that provides complete and actionable insight to help them address a wide range of issues across departments and lines of business. Unfortunately, enterprise data is constantly changing. Companies have to deal with tens of millions of daily changes in customers, partners and equity markets, along with high organizational and competitive fluidity and strict and evolving regulatory compliance requirements.
Staying on top of these changes and requirements is a tremendous challenge: just when one marketing database is cleaned up, a new campaign brings in all-new leads. As soon as one system migration is completed, another company is acquired and so on. To be successful in this environment, companies must build a strong foundation of people, process and technology to ensure data quality.
Although regulatory compliance is often cited as the primary driver for data quality initiatives, according to AMR Research, a global view of customers, suppliers and materials could save businesses $1 trillion.1 To obtain this global view, companies are looking to improve data quality across sales, marketing, human resources, procurement, manufacturing, distribution and finance.
Examples of how companies can use data quality to address issues beyond compliance include improving cash flow, customer profitability and manufacturing quality.
- One of the worlds largest freight carriers wanted to reduce days sales outstanding by reducing the number of invoices kicked back by customers for correction. The freight carrier was having difficulty correctly calculating shipping rates because rates are based on a customers aggregate volume, and many customers did business with more than one of the companys five autonomous business units - and each business unit used a different customer code in its freight management system. To resolve this problem, the freight carrier started an initiative to improve correlation between divisional data and higher level corporate codes.
- A large retail bank wanted to improve profitability of business customers. In order to get a more accurate and granular understanding of the factors driving profitability, the bank had to match customer records across deposit, loan, retirement and call center systems with D&B information. The need for detailed profitability analysis was the impetus to address data quality issues such as alternative spellings, abbreviations, historical addresses, name changes, trade names, incorrectly used fields and data entry mistakes.
- A global consumer appliance manufacturer wanted to reduce warranty costs and product returns. It started an initiative, Right the First Time, to eliminate introducing poor quality products into the market by sensing and responding to production nonconformance at the time of occurrence. Data quality was a critical component of this initiative because only the extensive analysis of data across multiple systems could determine whether poor product quality was caused by product design, raw material or production plant issues.
Creating a Foundation for Data Quality Success
Data quality is not a new concept, but has evolved over the years. In the 1990s, data quality was about consolidating data from multiple systems into a data warehouse to facilitate reporting and analysis. This was a very IT-focused project using extract, transform and load (ETL) and metadata management technologies for data profiling and integration. Once data warehousing projects were completed and companies started generating reports, they realized there were many errors, missing or incomplete data, and duplications in the source systems. In order to address the garbage-in-garbage-out problems of data warehousing, companies started new IT projects using cleansing and enrichment technologies.
Now companies have realized that in order to take data quality to the next level, they need standard definitions of business entities like customers, products, suppliers and employees. This growing awareness is being driven by data management issues such as mergers and acquisitions, information as a service and regulatory compliance. This requires master data management technologies to introduce formal data governance processes that are defined by cross-functional councils and implemented by data stewards who understand the business context of information. The most successful companies have moved data quality from an IT project to an ongoing business program with executive commitment and formal program funding as part of the corporate budget.
Executive sponsorship from senior management and business units is needed because data quality initiatives usually cross departmental and divisional boundaries. According to Gartner Inc., less than 10 percent of organizations will succeed at their first attempts at data governance because of cultural barriers and a lack of senior-level sponsorship.2 Executive sponsorship helps companies create a shared vision for the value of information, ensure program funding, and overcome political and behavioral barriers to success.
A governance council made up of people who represent the interests of the different groups affected by data management decisions is needed to define policies, standards, procedures and metrics relating to data quality. This group must have formal authority and accountability for data quality so they can resolve disputes, approve projects and enforce governance processes. Governance councils must include IT people who understand applications, security and data architectures, as well as businesspeople who understand corporate strategy, tactical processes and the data needed to drive business success.
Data stewards who are accountable for data quality are needed to execute the plan developed by the governance council. These individuals or teams are usually line-of-business subject matter experts and perform the day-to-day activities to ensure compliance with data quality criteria such as accuracy, completeness and integrity for a specific domain of data. Data stewards must have formal goals for data quality and incentives tied to those goals. Expecting data stewards to voluntarily ensure data quality in addition to performing their regular duties is a surefire way to sabotage the success of data quality initiatives. As data volumes grow, companies need to create workflows for distributing work across teams of stewards and ensuring the overall data validation and alignment process is under control.
Although there are many processes relating to data quality, they can generally be grouped into three categories.
- Analysis processes help companies understand the current state of data quality and the root cause of problems. Processes are needed to inspect the content and structure of data and provide detailed information about metadata, field formats, frequency counts, relationships and validity. Data governance is already being done in every organization, whether a formal program or not, so processes are needed to identify what practices are already in place and where there are opportunities to improve.
- Standardization processes help create accurate and consistent data. Companies need processes to establish and formalize elements such as data and business rules definitions, issue resolution and change management procedures, data owners, compliance and security policies, and quality metrics. Once data standards are established, processes are needed to cleanse, validate, match and link data across systems. Companies also need processes to communicate policies, standards, procedures, and metrics relating to data quality, along with processes to ensure employees understand them.
- Control processes help ensure long-term data quality across departments and divisions. Companies need processes to monitor data, check quality parameters, determine appropriate actions, and log detected faults and resulting actions. It is critical that control processes are implemented at the input sources because data entry is the most common origin of data quality problems according to the Data Warehousing Institute.3
To be successful adapting to changes in todays business world, companies need quality data that provides complete and actionable insight. The key to data quality success is building a strong foundation uniting people and process. A data quality initiative requires executive sponsors, cross-functional councils and subject matter experts accountable for data stewardship. Analysis, standardization and control processes are required to understand the current state of quality, create accurate and consistent data, and ensure quality over time. With the right combination of people and process, companies will be able to manage the entire data lifecycle and ensure they have the most accurate, complete and continuously refreshed information for ensuring success in todays faster, global and more scrutinized business environment.
- Bill Swanton. Master Data Management Framework: Begin with an End in Mind. AMR Research, September 2005.
- David Newman. Governance Is an Essential Building Block for Enterprise Information Management. Gartner Inc. May 2006.
- Philip Russom. Taking Data Quality to the Enterprise through Data Governance. The Data Warehousing Institute, March 2006.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access