Information Integrity

The Next Frontier for Data Warehouse Managers

InfoManagement Direct, February 8, 2008

Steve Linsley, Angsuman Dutta

Despite millions of dollars worth of investments, information within the data warehouse continues to be inaccurate, incomplete and often inconsistent with its sources. As a result, data warehouses experience low confidence and acceptance by users and consumers of downstream reports. Additionally, in many cases data warehouse projects have failed.

Organizations today face many of the following challenges related to the management and control of data warehouse information: accelerating changes in the business environment, increasing complexity of source systems and technology, and an expanding array of regulations and compliance requirements such as Sarbanes-Oxley (SOX) and numerous industry-specific regulations. The integrity of the information in most data warehouse projects today is often unclear, ill-defined and suspect. It is necessary to have definitions and standards of information integrity and deploy an effective system of automated information controls to monitor, measure and maintain information integrity throughout the data warehouse to meet business intelligence (BI), performance management, compliance and the increasingly high demand to execute at unprecedented levels of operational excellence.

Organizations are operating in complex environments. They constantly generate, use, store and exchange information and materials with customers, partners and suppliers. Enterprises are also required to exchange key performance information with regulatory agencies and shareholders. Today’s connected enterprise can only be as its information quality. Trustworthy information reduces risk and uncertainty in the decision-making process, enhances confidence and improves operational effectiveness.

Today, many companies use data warehouses to centralize and organize information from which reporting then lead to major business decisions. In fact, Information Week reports, “data warehouses are the critical underpinning of business intelligence projects. And companies have big plans for business intelligence.”¹ In addition, many companies also rely on data warehouses to support their key compliance programs and numerous other information management initiatives.

Companies have spent millions of dollars to build data warehouses. A simple warehouse may merely take data from several upstream sources and consolidate it. A more complex data warehouse may aggregate and combine data from disparate systems, lines of business, business partners and more - into a single warehouse. No matter the size or complexity, the quality of the information that a data warehouse contains is critical to determine and manage.

Bad information decreases operational efficiency and leads to dire drills, rework, bad business decisions, and, in some cases, embarrassing headlines. Even worse, it has lead to material misstatements in financial filings. In fact, the Data Warehousing Institute estimates that companies lose more than $600 billion every year due to bad information.² This includes employee time, lost customers and lost revenue. A data warehouse may fail if users do not trust the quality of the information it contains.

The effect of bad information is worse for companies that need to use information from a data warehouses to meet the requirements of SOX, SAS 70 audits and Basel II. In these cases, if a data warehouse does not provide accurate, consistent and reliable information, a company and its leaders can face fines and penalties. For these reasons and more, the integrity of information that a data warehouse contains is a critical issue for every company.

The Data Warehouse Challenge

The accuracy, completeness and timeliness of the information present in a data warehouse are often questionable. In the past, data warehouse managers have been able to provide reasonable assurance as to the quality of the data because they either had the time to inspect and correct issues, processes, and controls or those problems were sufficiently isolated to prevent wide-scale contamination (due to the loosely coupled or discontinuous and physical nature of the data in the system at that time). However, several recent trends in the business environment are dramatically limiting internal auditor’s abilities to provide assurance. The urgency of finding new approaches to this issue is increasing.

Data warehouse managers face many challenges: accelerating changes in the business environment, changing needs of the business users, increasing complexity of systems and technology and an expanding (and ever-changing) array of regulations and compliance requirements.

Accelerating Changes

Every organization faces internal and external changes. There may be various reasons for these changes - new products and markets, restructuring, personnel turnover, new information systems or changes in the regulatory environment.

Experience shows that due to changes in the organization or its external environment, controls implemented may no longer adequately address current risks.

This means having information systems in place to ensure that all the relevant information is gathered reliably and at the right time, and distributed to the right people.

Increasing Complexity

In past decades, organizations grew as a result of innovation in products, processes and systems and expansion of geographical boundaries of operations. One of the consequences of all this activity has been an enormous increase in the complexity of their businesses, which tends to increase the fixed costs of conducting their business. This complexity manifests itself in many forms, affecting everything from the day-to-day BI operations data warehouse to strategic business plans. For example, a bank uses several dozen different applications to support its deposit account process. It is impossible to manually reconcile the deposit account information stored in a data warehouse back to its source systems due to the vast volumes of transactions. The complexity within business enterprises increases with time. Adding to this complexity are several factors including: higher volumes of data, faster rates of processing, more types of data, more hardware and software platforms, and more interfaces. Every new interface, every new component and every new feature adds to the complexity and increases the potential for errors.

Regulations and Compliance Mandates

As companies comply with the reporting requirements of Sections 302 and 404 of the SOX, various SAS 70 requirements, the Patriot Act, Basel II and varied service level agreements (SLAs), data warehouse managers are challenged with documenting, testing and certifying controls that ensure the integrity of the financial and operational information present in the data warehouse. In addition, internal auditors and external auditors are faced with proving the accuracy and the completeness of the financial information present in the data warehouse. According to Internal Auditing magazine, “Rapid advances in hardware and software innovations and companies’ integration of data warehouses create a dramatic change in the technological control environment, directly affecting the financial reporting system. The advancements, in turn, require the internal auditor to adapt and develop specific auditing procedures.”³