During the data warehouse boom of the 1990s, there was often tongue-in-cheek talk that the data warehouse was the "Full Employment Act for database administrators (DBAs)." This was due to the explosive demand for DBAs to wring performance out of database management systems (DBMSs) that were not built for data warehouse functions.

Don't look now, but another "full employment" simile may be in the works. The reasons will have nothing to do with technology but with regulatory pressure and economic changes. I am talking about an increase in data administration and data management jobs. In addition to companies realizing that information asset management (IAM) is now a necessary part of business (see July 2003, DMReview.com article on convergence,), public companies and others are confronted with a rather complex series of legislation that are, indeed, forcing information management issues to be addressed. A savvy CIO must be proactive in light of these changes.

Examples of these legislative catalysts to data management abound. HIPAA, Graham Leach Bliley(GLB), Sarbanes- Oxley (SOX), etc. all present organizations with a legal reason to achieve elevated levels of information governance. Some of these regulations are specific (e.g., HIPAA and privacy and encryption) and some are obscure (SOX), but all are pretty clear in their intent - if an executive does not effectively manage the quality, accuracy and content of the organization's information, the organization and its executives are subject to fines and prison.

After reviewing (and, yes, reading) several of these laws, a few crucial themes are emerging. We will visit those themes later, but it is key to note that all of these themes are dependent on managing information. The consequences of failing to do so result in fines and jail time - serious considerations. The bottom line is there must be information governance. It is no longer an option. A data warehouse, Corporate Information Factory, meta data awareness, etc. represent an initial start, but there must be a lot more.

Visibility into Controls

A good data warehouse has effective controls for ensuring data integrity as do good finance systems. Rarely do the two sets of controls meet up. When they do, companies can actually use the DW for financial reporting. This is beginning to happen with greater frequency.

However, controls must now be well documented and tie to each other. (SOX Section 404). This means possible reengineering of controls for applications. Since controls are also business rules, this also means meta data needs updating. It is no longer enough to say "the data adds up."

During a recent speech, I asked a crowd of about 150 people, "Who can define for me a hash total process?" Very few hands were raised. While this was a crude survey, I submit that data administrators may have to become familiar with basic accounting controls and create meta data to support and define these. Across SOX and the other regulations, companies must prove they know what they know.

Tracking of Changes

Many regulations require tracking of changes to data. Like the controls requirements, this means meta data to track changes in dimensions and attributes as well as monitoring changes to actual data values. Adjustments to static data have been a pox on data warehouse architects. Normally, someone is allowed to "tweak" data so accuracy can be assured. This is now, potentially, an illegal act. Adjusting data must now be controlled via rules and authorizations - all meta data requirements.

Privacy and Encryption

Several of the regulations require encryption, removal of SSN as primary key and mandate clean data. HIPAA (and related state regulations, such as exist in California) mandate that SSN is never to be used as an identifier for an individual. Additionally, individuals health records cannot be identifiable by knowledge workers performing aggregate analysis.

GLB requires stringent controls on personal financial information. While seemingly a regulation that targets only banks, it affects any company that issues credit, e.g., consumer product, automobiles, builders, etc.

Again, at the risk of repetition, the management of all of this depends on meta data.

Technology Trends

All of the above implies a robust meta data layer. At minimum, some type of passive catalog that contains not only the traditional entitles and attributes, but also rules, context and, heaven forbid, documentation of the actual reports and metrics to be reported. There are no tools or repositories that explicitly offer these elements. While the top repository offerings can be extended, there are few examples and even less experience of successful use of repositories to monitor controls, context and metrics.

The technology challenge compounds when these same, aging tools (traditional repositories) must also adapt to low-latency data handling (information buses, etc.). The Internet is that data management areas will need to adopt some type of basic repository engine and then blend in some homegrown extensions.

Culture Changes

Perhaps the most significant change in store for IT will be the fundamental changes in how data administrators (DA) and DBAs execute their tasks. DAs will need to learn fundamentals of accounting controls. DBAs will need to carefully monitor access to data values. CIOs will need to pump money into both of those areas in the form of training, tools and external consultants to ensure continuity and compliance.

Training must come from areas familiar with the laws in detail. Consultants must be more than good meta data practitioners or data warehouse designer. I submit that at some point, independent auditors may need to certify IT consultants (not the lame self-promoting certifications that proliferate in our industry).

Data administration is an area that, in many companies, has been either a programmer retirement zone or a misunderstood and, therefore avoided, function. Now this group of people has the skills to keep the CxO away from jail. This is a commendable responsibility, but the CIO must confirm that the existing data management groups can handle the accountability and move to reinforce their capabilities.


In addition to architectures that manage the actual data (warehouse, CIF, etc.), there will need to be architectures that manage the meta data. There will layers of architecture that feature controls and business rules. There will be a supervisory component for compliance that can intrude into every aspect of data handling to ensure accuracy, privacy, control and whatever else regulators have in store.

Figure 1: Layers of Architectures

All of these requirements go beyond mere tools and a few procedures. Historically reliable techniques and tools may prove totally inadequate. Therefore DAs, DBAs and CIOs must be prepared to be creative, focused and accountable to their business sponsors. Meta data is no longer a nicety, but a crucial component of enterprise technology architectures.

The contents of this article are Copyright 2003 by DM Review and KI Solutions. Any use, quotation, repurpose, duplication or replication of the diagrams, concepts or content without permission of DM Review and the author is prohibited.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access