When an enterprise first encounters master data management, it often doesnt have a clear understanding of how MDM will affect the architecture of its business transaction systems or business intelligence systems. This article describes master data patterns in legacy system architectures, a general MDM architecture and some ways the new MDM layer affects the master data patterns in the legacy layers.
Master Data Patterns in the Business Transaction Layer
Master data patterns in the business transaction layer of the architecture follow the lifecycle of master entities:
- Systems that originate master data,
- Systems that update master data,
- Systems that expire master data and
- Synchronization of master entity state changes across systems.
Points of Master Data Origination
A business transaction layer can have one or more systems that create a master data entity. For example, a large bank can have 50 to 100 product systems that must create a customer entity to establish an account. In contrast, a large manufacturing company, even though it has multiple product lines, may have a single system of customer creation in an enterprise resource planning system. If, however, a manufacturing company has not integrated its order entry system with its service system, there may be a second point of customer creation in the service system. In business transaction layers with multiple origination systems, I typically see master data duplicates spread throughout the system population. Occasionally, single origination systems can also have duplication if poor data controls are in place.
Points of Master Data Updates
After origination, master data migrates downstream to other systems that use it. For example, a manufacturing company may originate part data in its engineering system, which passes the data to systems that support manufacturing, inventory and sales. These downstream systems may change master entity attributes or add new ones that reflect their viewpoint on the master data. For example, an inventory system may add financial classification codes to Part data for asset valuation. In business transaction layers with multiple systems that update master data, we typically see master entities in multiple, simultaneous states which are often inconsistent. The problem can compound if there are multiple levels of dependence. For example, system A updates ABC; A sends ABC to system B; B updates ABC; B sends ABC to system C.
Points of Master Data Expiration
While master data might never be deleted completely (archived instead), its value and meaning can expire. For example, an angry customer can close an account and their relationship to a service enterprise. Or manufacturing company will withdraw products (and service/support) from its markets over time. When these business events occur, the business transaction system layer should reflect this state change in its master data. In business transaction layers with multiple points of entry or update, we typically see master entity duplicates in simultaneous, inconsistent states of expiration.
Master Data Synchronization
Organizations are well aware of these business transaction master data patterns. In the old batch days, enterprises sometimes had circular file systems that mimicked the manual file systems they replaced. Today, some organizations have data hubs that implement the same concept and attempt to synchronize master data between points of origination, update and expiration. Other organizations have a spaghetti-like network of point-to-point synchronization.
In business transaction layers with multiple points of entry, update and expiration, I typically find some form of synchronization implemented in a range of complexity and effectiveness. One problem that confronts these solutions is their ability to change versus the rate of system change (i.e., new systems, system updates and system retirement).
Master Data Patterns in the BI Layer
Master data patterns in the BI layer of the architecture follow the process cycle of business intelligence:
- Master data collection,
- Master data integration,
- Master data reporting and
- Synchronization of master data state across systems.
Points of Master Data Collection
A BI layer can have one or more systems that receive a master data entity from the business transaction layer. For example, a bank could have a large data warehouse for its commercial business and a large data warehouse for its consumer business. Product systems in each business area feed customer data into these warehouses. In other cases, a business transaction system may bypass the organization data warehouse(s) and send customer or product master data directly to a dedicated data mart.
In BI layers with multiple collection points, I typically see master data spread across BI targets - usually a combination of warehouses and marts. If the timing of source data into collection systems is different, or the source data is inconsistent, then the master data may be in an inconsistent state across BI targets. Even when there is a single collection point, we can see master data duplicates if there are weak data controls in the collection system; the master data might then spread across a population of downstream data marts.
Points of Master Data Integration
A BI layer can have one or more points of integration of entity master data from the business transaction layer. For example, a bank with a large data warehouse for its commercial business and a large data warehouse for its consumer business might have separate integration systems for each warehouse. In other cases, there may be a hierarchy of data marts where the child tier data marts integrate master data from different source data marts in the parent tier.
In BI layers with multiple integration points, I typically see inconsistent combinations of master data in BI targets. The problems of different collection times or inconsistent source data are compounded by inconsistent integration. Even when there is a single integration point, master data duplicates can integrate inconsistently as they move through levels of data mart hierarchies.
Points of Master Data Reporting
A BI layer typically has many downstream points of reporting that use master data. For example, a bank with a large data warehouse for its commercial business and a large data warehouse for its consumer business will usually have numerous downstream data marts producing analytic reports. In addition, there will often be many departmental reporting systems implemented using spreadsheets sourced from data marts.
In BI layers with multiple collection and integration points, I typically see inconsistent master data across the organization reports. Key metrics and statistics will not reconcile. Executive reports that cross vertical dimensions of the BI layer often require teams of staff to manually adjust data off the source reports.
Master Data Synchronization
Organizations are well aware of these BI master data patterns. Periodically, organizations attempt to consolidate multiple points of collection, integration or reporting into single functional points. Sometimes these efforts encounter problems of scale. Even when these efforts succeed, scale and adaptability are often inversely related. Departments will seek ways around centralization and re-initiate data warehouse and mart proliferation justified by expediency.
The MDM Layer
An MDM layer in the architecture generally has the following components:
- Master data repositories,
- Data quality tools,
- Data integration tools and
- Data synchronization tools.
Master data repositories. Master data repositories store the golden instances of master data. In addition to the master data entities and attributes, the repositories can contain important alternate identifiers, codes and relationships. Data quality. MDM layers usually contain tools that can read master data from business transaction or BI sources and evaluate whether or not the master data conforms to standards based on key business rules. The systems typically have the capacity to periodically monitor business transaction or BI sources and produce alerts when noncompliance occurs.
Data integration. MDM layers also contain tools that reconcile master data from multiple business transaction or BI sources into single golden instances. In some cases, these systems also use specialized data structures to enrich or enhance the integrated master data.
Data synchronization. MDM layers also contain tools that collect and communicate master data state changes from/to sources systems in the BI and business transaction layers. These systems typically contain mapping metadata to translate the MDM version of the master data into different forms that may exist on the business transaction or BI source systems.
MDM Impacts on Legacy Architecture
The addition of an MDM layer to an existing architecture has the following possible impacts:
- A new relationship between the business transaction and MDM layers,
- A new relationship between the BI and MDM layers,
- Modifications to the legacy relationship between the business transaction and BI layers and
- New data quality visibility in the business transaction and BI layers.
The Business Transaction-to-MDM Relationship
The presence of MDM in the architecture introduces a new interface to the business transaction layer. This interface is both inbound and outbound to business transaction systems.
The outbound interface from business transaction systems affects all points of origination, update and expiration. For example, an origination point must now notify MDM when it creates a new instance of a master entity. Likewise, an update point must now notify MDM of changes to master data attributes or relationships.
The inbound interface to business transaction systems affects the synchronization of points of origination, update and expiration. For example, a new entity created by an origination system may already exist; if so, MDM can send a transaction to synchronize the entity in the origination system. In another case, a system may deactivate an entity that must be kept active overall; if so, MDM may send a transaction to reinstate the entity.
MDM synchronization may conflict with existing synchronization mechanisms. For example, data hubs pass both master data and nonmaster data to their client systems. Inbound MDM interfaces duplicate the data hub processes for master data. In cases where there is a point-to-point synchronization network, organizations may have to reorganize the network and its content.
MDM introduces new governance relationships into the business transaction layer. For example, systems introducing new master data through development must coordinate its changes with changes in the MDM layer. In addition, when the enterprise introduces a new system to the business transaction layer or consolidates systems in the business transaction layer, this activity requires coordination of changes to the MDM repositories and interfaces.
The BI-to-MDM Relationship
The presence of MDM in the architecture introduces a new interface to the BI layer. This interface directly affects points of collection and integration and indirectly affects points of reporting.
The interface to the BI layer affects points of collection in two ways. First, organizations with multiple collection points may choose to use MDM as a single source for master data. Second, regardless of its source role, MDM will play a validation role for master data in business transaction transaction data flowing through the BI collection points.
The interface to the BI layer allows organizations to reduce or eliminate multiple points of master data integration. In cases where rationalization is not possible, the MDM layer can play a validation role on data warehouse or data mart integration points of master data.
Master data synchronization is now possible from a central place in the architecture (MDM layers can typically handle large scale). To the extent that points of collection and integration are under MDM controls, consistency between BI reports generated from different chains of these collection and integration points should improve.
Some organizations create master data in the BI layer. In these cases, business transaction-MDM interfaces will also execute for the BI layer. For example, a marketing department might maintain product master data in a warehouse or a data mart with associated point-to-point synchronization in the BI layer (and sometimes to the business transaction layer). Add, change and expire state changes to this master data will need to flow to the MDM layer for data quality, integration and syndication.
MDM introduces new governance relationships into the BI layer. For example, systems introducing new master data through development must coordinate its changes with changes in the MDM layer. In addition, when the enterprise introduces a new system to the BI layer or consolidates systems in the BI layer, this activity requires coordination of changes to the MDM repositories and interfaces.
The BI-to-Business Transaction Relationship
The presence of MDM in the architecture alters the legacy master data integration points established in the business transaction and BI layers. In data management literature, MDM that is focused on the business transaction layer is called operational MDM; MDM that is focused on the BI layer is called analytical MDM.
When organizations implement operational MDM, master data integration occurs before it enters the BI layer. The burden on the BI layer to collect inconsistent master data and integrate is reduced. (Not all master data may be within the scope of operational MDM, especially in the initial implementation phases.) Effectively, business transaction sources of master data disappear and are replaced by MDM; in addition, the timing of master data integration is different. These modifications due to MDM then affect collection and integration points in the BI layer.
When organizations implement analytical MDM, the business transaction layer origination, update and expiration points dont change. Instead, the MDM layer collects and integrates master data for use in the BI layer. Interfaces that connected business transaction points of origination, update and expiration to BI will now connect to MDM; functionality in the BI layer used to integrate business transaction master data becomes redundant.
MDM also affects governance relationships between business transaction and BI. For example, new master data introduced into the business transaction layer by new systems will coordinate these changes with the MDM layer; coordination with the BI layer occurs through the MDM/BI relationship. Master data changes that occur via system development or consolidation also follow this new pattern of coordination.
Data Quality in the BI and Business Transaction Layers
The presence of MDM in the architecture creates a new locus for master data quality that is independent of business transaction or BI system interests. The data quality tools in the MDM layer can profile business transaction or BI data structures and quantify the degree of compliance to master data standards. MDM analytics can pinpoint the location (and thus responsibility) for master data problems in legacy system programs, business processes or both. Once MDM has established data quality measurement, an organization can set goals for improvement and initiate work efforts to modify systems, business processes or both.
When MDM comes to an organization for the first time, a lot of assumptions about what it is and its effects. In fact, the effects of MDM on an existing architecture can be broad and deep. By knowing the high-level impacts of MDM on an existing business transaction/BI architecture, you can preempt misunderstandings that, down the road, affect your MDM implementation and the business value and cost of improving the quality of your master data.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access