Related Links

Web Seminars

The thought of meta data management doesn't exactly inspire whoops of joy from the average IT manager. However, like flossing, it's one of those things you've just got to do - and for good reason.

Done right, meta data management makes it easier to manage and understand data - an especially good thing in this age of Sarbanes-Oxley. It improves data quality and makes it auditable. So why hasn't everyone done it?

Let's divide the world into three camps regarding meta data: those who ignore it, the zealots and the clueless. Plenty of companies have devoted significant time, resources and money to meta data management with the zeal to make it successful. However, for the clueless ­ which is most companies ­ meta data management needs to start small, produce value and then expand. For these firms, where do you start?

Data About Data

The classic definition of meta data is data about data. Meta data is the description of the data as it created, transformed, stored, accessed and consumed in the data integration framework. Businesspeople need to know what the data represents - where it came from, how it was transformed and what it means. IT people need to know what happened to the data from the point of capture through its consumption by the business in their reports and analysis.

There are two types of meta data: technical and business. This distinction has been poorly understood and has caused much confusion, especially with IT and software vendors. Technical meta data is the description of data as it is processed by software tools. Databases, for example, need to define columns (format, size, etc.), tables and indexes; extract, transform and load (ETL) tools need to define fields, mappings between source and targets, transformations and workflows; and business intelligence (BI) tools need to describe fields and reports. All of this meta data is used to enable the software tools (not people) to understand and process data.

Business meta data, in contrast, is the description of information from the business perspective (e.g., the business context of the inventory turns, weekly sales or budget variance reports). Some of the meta data that describes a report is technical - such as field size and type. However, most of the data the businessperson cares about is not used by the software tools. BI tools have implemented "semantic layers" where text can be associated with fields to allow the input of business descriptions. While helpful, this is not nearly extensive enough to encapsulate the full business meta data needed.

It's Dirty Job ...

Managing meta data can be a dirty job, but a few guidelines can smooth out the process.

First, manage the scope. If you try to "boil the ocean" you will get burned. Unless budget is not an issue, constrain your goals in the beginning. This will keep you from spending too much time and money, wasting resources and delaying the project. You will probably only get one chance, so start small.

Second, address the cultural and political issues. Know who owns the data, who has time to define it and if management really is committed to the project. It is nice to get everyone excited about meta data management, but is it on someone's priority list? Is it considered someone's real job?

Establish standards and processes that ingrain meta data management into business and IT people's jobs. Standards and processes need to be part of what we do, no excuses. The incentive of Sarbanes-Oxley and other regulations should spur management to back a reasonable meta data initiative.

Finally, don't count on technology to be a silver bullet for meta data management. It's dirty, it's ugly and you typically need to piece together a solution. In addition, a lot of the work involves talking with businesspeople about what the data means and finding where the meta data is. Don't be scared away. Keep the scope small enough that your technology can handle it without significant custom software development.

Six Levels of Meta Data

There are several levels of meta data that can be managed:

Data element descriptions: Documenting data element descriptions should be part of the requirements gathering of any BI or data warehouse (DW) project. As such, you have already done the heavy lifting for this type of meta data. This is where most meta data initiatives start, and typically a data dictionary is created. Although the data elements are initially documented in the project's business requirements document, they are usually stored and queried from a data modeling tool or BI tool accessing a customized data dictionary stored in a relational database. More sophisticated projects may have purchased and deployed a meta data repository.

ETL source-to-target mapping: If you have used an ETL tool rather than custom code, then this should be an easy step. Your ETL tool should have a catalog that can be used for documentation and for examining the ETL workflow. Data lineage, at least within the context of your ETL code, can be examined and queried both to determine how a data element was transformed, as well as what the impact of a change would be on that data element or workflow. This meta data, although of interest to business, is generally an IT tool and improves their productivity.

BI query cataloging: This elementary meta data management step includes BI query cataloging of reports, pre-built queries and data element definitions. Most of today's top tier BI products come with BI catalogs that store information and allow queries from BI query tools. In addition, portals are also available within these BI product suites that allow grouping of reports, queries and data elements that business users can access.