More Data Quality for Confident Operational Business Intelligence

Published
  • March 01 2010, 10:08am EST

In my last column, I described two key implicationsof operational (versus traditional) business intelligence - source data quality and governance/data stewardship. These are necessitated by the processing cycles for operational BI, which do not allow for batch processing and error correction. The two implications discussed in this column are continuous quality improvement and metadata capture and dissemination.

Continuous Quality Improvement Cycle

Continuous quality improvement is a concept introduced by Dr. Shewhart in the 1930s and popularized by W. Edwards Deming as a critical component of any effective quality management program. The continuous quality improvement cycle as shown in the figure emphasizes the importance of planning a task, performing it, measuring the performance, taking appropriate corrective actions based on the measurements and then repeating the process. In operational BI, this concept recognizes that unless we address problems at their source, they will recur, which means that erroneous data will be introduced into the BI environment.

Even with the most thorough data profiling (discussed in my previous column), new data quality deficiencies are likely to arise. For each deficiency, the continuous improvement approach dictates that the root causes - not just the symptoms - of the problem are addressed. Symptoms represent the visible problem (e.g., incorrect or incomplete data); causes represent the reasons that the error occurred. The root causes are the most significant contributing factors to the deficiency, hence, they should be addressed to eliminate or substantially reduce the deficiencies.

While many tools are available to support root cause analysis, it is a difficult process. First, it requires a candid examination of the environment to trace data from its origin and determine how an error occurred. Some of the difficulties include: The existing environment may not be set up to trace data throughout its lifecycle; there may be multiple causes, and determining the root cause may be difficult and time-consuming, particularly if empirical data to support the analysis is not available; and the causes often entail operational systems, business process, worker education and worker proficiency, skill and motivation.

Root cause analysis, by itself, does not eliminate the problems. Action to address the root causes is required to ensure that quality data flows to the operational BI environment. Perform the analysis to determine the appropriate action, and then pursue it with the results measured. The governance structure previously described is critical, particularly if the changes entail business processes and personnel. The continuous improvement cycle also dictates that companies institute suitable data quality measures, track them over time and take appropriate actions when problems occur.

Metadata Capture and Dissemination

Both traditional and operational BI users require legibility. They need to understand the data that they are using so that they can make better, more informed decisions. With traditional BI, the business process the analysts are performing is decision-making (in the context of a business domain). The primary job of many of these business users is business analysis, and they often have special training on how to do that. Because they are using BI information delivery tools, metadata delivery should be provided in the context of those analyses with those tools.

With operational BI, the line of business process is being performed and information is being provided to help the operator, analyst or manager understand a situation and trigger an action. The primary job of each of these individuals is to perform or manage a business process or operation. These people are often experts in their respective fields, but they may not have specialized training in business analytics and in using the data that BI can deliver to manage and improve their operation. They may require a richer set of metadata to help them find the data and to ensure that they thoroughly understand the data they receive and how it can (or cannot) be used.

Operational BI is the wave of the future. Many companies have adopted it and others will soon follow. People must be able to trust the data if they are expected to use it. In addition to providing a formal process for eliminating the root causes of data deficiencies, the continuous improvement cycle introduces measurements to help people understand the quality of the data so that they can make informed decisions on its use. Those measurements can be provided to the business users along with other metadata to ensure that they have the information they need to effectively use operational BI capabilities to improve the business.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access