In the first part of this two-part series on data assurance, I walked through the stumbling blocks that plague many data assurance initiatives. In addition, I discussed the first three phases of the data assurance road map:

  • Set/Refine Objectives: What are the business objectives?
  • Data Assessment: Where are we currently?
  • Data Alignment: Where are we going (end state)?

In this second installment, I will address the final three phases of the data assurance road map:

  • Data Harmonization: How do we get there?
  • Solution Implementation: Implement the processes.
  • Continuous Improvement: Feedback to refine and extend.

Figure 1: Data Assurance Road Map

Data Harmonization. Data harmonization is the process of mapping the "current" state of our data (as defined in the data assessment phase) to the target "end" state of our data (as defined in the data alignment phase).

A managed meta data environment (MME) is the key application in the data harmonization phase (see Figure 2). The MME is responsible for the enterprise-wide meta data management and utilization. The IT staff and data stewards will directly interact with the MME to perform the process of mapping the current data state to the end data state. This process is vital in many applications as it allows an organization to transform their data into a single version of the truth.

Figure 2: Managed Meta Data Environment

Solution Implementation. The goal of the solution implementation phase is to physically build the processes and procedures to achieve the data harmonization. During this phase, several items must be considered.

First, as bad/inaccurate data is identified, it must be tagged and placed into a database for reporting purposes. Second, the business users must be actively involved in the process of examining the inaccurate data so that feedback can be provided back to the data's source.

Third, often data will error-out because the business rules for the data are not complete or correct. At this point, the business users should be able to identify these situations, and the process must be modified for future data extractions. Lastly, records that error-out should be updated as new data becomes available or if the source record itself has been updated. In these cases, the records would then recirculate back into the process. In the case of a data warehouse, these previously tagged inaccurate records would then be marked as "clean" and subsequently loaded into the data warehouse.

Continuous Improvement. Data assurance is an evolutionary process. Therefore, organizations need to have an ongoing feedback mechanism and update capability to modify and improve its performance. Meta data and proper meta data management is the key to this initiative as it supports all of the six phases discussed here. For example, all of the information collected during the monitoring of an ETL process is meta data. This meta data needs to be persistently stored and historically tracked within the MME so that the IT staff and data stewards can leverage it to accomplish each of the phases of the road map. For a greater discussion on meta data management, please read Building and Managing the Meta Data RepositoryUniversal Meta Data Models (David Marco, Wiley 2000) and (David Marco & Michael Jennings, Wiley 2004).

When properly implemented, data assurance can greatly aid a company's ability to adapt their systems to change and to deliver strategic and tactical information to their business users in business terms. Keep in mind that "easy" is the one word you will not hear in connection with implementing data assurance. To be successful in your data assurance program takes knowledge, discipline, talented employees and good old-fashioned hard work, just like any other major IT endeavor.

The quality of information that flows through the enterprise applications, databases, networks, business processes and business rules, external processes linking to customers/suppliers/partners, external market knowledge and syndicated data sources is extremely critical in not only the day-to-day operation, but also in making tactical decisions and setting strategic direction for the enterprise. In order to enable the highest level of information integrity, we must first assess the level of quality of information at each layer -- not just the technology layer, but also the process and people layer. While data assurance is not a silver bullet, it provides a framework to avoid costly mistakes and disappointments when using data integration technology.

This column is based on the "How to Design and Deliver an Effective Data Assurance Solution" white paper. Please visit to download a free copy of the paper.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access