An increasing number of our clients are asking, "What is this data?" Sure, the business intelligence (BI) applications they are using are providing data, but they want to feel comfortable with the origin of the data, the business rules that were used to manipulate it and want the context in which the description of the data elements to make sense. In these situations, our clients are asking about the end-user presentation of meta data. Simply defined, meta data is the data definition of data. There are two primary levels of meta data: administrative and end-user layers.
The administrative layer of meta data relates to the application and the database configurations that are stored in entities within a database. While this is an important layer, the end-user layer of meta data is where most of the questions and differences of opinion arise. Figure 1 attempts to define and classify the various levels of end user meta data.
|5||Report data||Within a report, data is manipulated into a format that is required.|
|4||Business intelligence data||Data which has been extracted by a BI application. This data may be manipulated using business rules.|
|3||Transformed data||Data that has been transformed using business rules into a format that is meaningful and is stored within a database.|
|2||Cleansed data||Data which has been processed through a cleansing routine which standardizes the format and is stored within a database.|
|1||Migrated data||Data which has been migrated from several source systems into one database and is commingled.|
|0||Source system data||Data which has been entered or processed by an online transaction processing application.|
While these levels define the transgression of data from its origin to a report, the complexities of each level can have a significant impact on the definition of the data. Please note that not all levels of end-user meta data are encountered by an individual using a BI application. Levels 1 - 3 relate to the data warehousing of information, in which case an individual lacking access to a data warehouse may encounter only levels 0, 4 and 5 when extracting data directly from a source system.
Source System Data
In each transactional system, a user of the system must enter data. For example, a sales representative enters information about a product that was sold to a customer within an order entry application. The information that is entered by the sales representative is accepted by the application and stored within a database. The data elements of information relating to the sales transaction may include the customer name, shipping and billing addresses, terms of the transaction, product purchased, quantity sold, sales price and discounts.
Assuming that an organization has multiple transactional systems and the desire to warehouse the data, migrating the data from the source systems to an operational data store or a data warehouse entails the identification of the source system as the data is commingled. For example, customer information is migrated from the order entry and the customer relationship management application into a single database used as either an operational data store or the staging area of a data warehouse. The data elements of information relating to the customer information may include customer name, tax payer identification number, customer type, contact name, address and phone number from each of the source systems.
After the data has been migrated to a central database, a cleansing routine can be applied to standardize the format of the data and validate the quality of the data. For example, customer information has been migrated from several source systems. As part of the cleansing process, data may be standardized and redundancy eliminated. With one of our clients, we noted that one customer name was uniquely defined in each of the source systems and that the redundancy was not eliminated until the cleansing process was conducted. In this example, imagine one customer with the following uniquely defined names: base consulting group, Base Consulting Group, BASE Consulting Group and BASE Consulting Group, Inc. The cleansing process would eliminate the redundancy, standardize the format and validate the data so that one customer name would remain: BASE Consulting Group, Inc.
Business rules and definitions are used to transform the cleansed data into meaningful information. For example, revenue may be defined as the quantity sold multiplied by the sale price less any discounts. This transformation of data derives a new data element from the data created by the source system(s). Defining the business rules as well as achieving acceptance of the definition can be challenging, and it is the level that encounters the most difficulty when trying to achieve consensus.
Business Intelligence data
Most BI applications can access data from a variety of sources including databases, ASCII files or spreadsheets. Additional transformations or masking of data can be directed by the BI application. For example, a data element called segment1 that contains information about purchase order numbers can be masked and renamed to PO Number. From the end user perspective the data element is called PO Number, but the database name for it is segment1, which does not provide a user with a descriptive understanding of the type of data that is associated with it.
Within a report, data can be derived that further enhances the meaningfulness of the information. However, this is the last meta data level within the end user layer. Data derived at this level typically relates to calculations such as subtotals and grand totals or conditional statements such as, "If gross margin % >= 50% then Highly Profitable Product’ is listed in the column next to the gross margin %."
As users embrace BI technology for analysis and reporting, they want to know what the data is so that they can put it into the appropriate context. By understanding the various levels of end user meta data, users can fathom the metamorphosis of data from its origin to the report.
LeBaron, M. and Adelman, S. Meta Data Standards. DM Review magazine. February, 1997.
Inmon, W. Enterprise Meta Data. DM Review magazine. November, 1998.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access