Continue in 2 seconds

Incremental Master Records - A New Approach to Enterprise MDM

  • October 01 2007, 1:00am EDT

Data management professionals face critical challenges in unifying enterprise-wide information. The plethora of mergers, corporate restructures, disparate lines of business and other means of corporate fragmentation have created the need to build and deploy new, composite and analytical applications that deliver the "right" answer. Conflicting values, incomplete records and the inability to create a single version of the truth have made it nearly impossible to create highly trusted master records across multiple applications, data warehouses and reference data sources. The need to eliminate "garbage in garbage out" as standard operating procedure is now more urgent than ever.

Enter Enterprise Master Data Management

Master data management (MDM) is a proven technology that provides infrastructure capabilities that unify disparate data sources into a uniform and enterprise-wide trusted source of truth. MDM, or its most common use case, customer data integration (CDI), is now an accepted technology framework for creating 360-degree views of customer, product and supplier. MDM is now often seen as a required component of new composite applications for both transactional and analytical use. There are two generations of technology approaches today for enterprise-wide MDM and CDI solutions.

The first generation of MDM focused on centralizing master data into a single physical data repository. Deployment of these solutions is typically driven by organizations seeking to create a universal transaction infrastructure, not necessarily by a specific business initiative. A second generation of MDM, emerging today, focuses on building enterprise-wide master records in a more pragmatic and incremental fashion. To understand the benefits of the second-generation approach, we need to start by taking a look at first-generation MDM.

First-Generation Enterprise MDM

A Single Transactional Database

First-generation MDM solutions attempt to unify all existing application databases into a single, master data repository by physically bringing together the databases from customer relationship management (CRM), enterprise resource planning (ERP), sales force automation (SFA), service and other production systems into a single transactional database. All enterprise applications would access this master database via either a direct interface or, more likely, Web services calls.

Reality Check

The benefits of a single, unified master data repository are obvious. If all enterprise master data is kept in a single repository, it would eliminate the fragmentation problems caused by disparate data that is inherent in today's distributed business environment. Despite its apparent value, however, several obstacles prevent organizations from pursuing such a model.

Unifying all enterprise master data from current production databases into a single universal transaction store and single schema essentially undoes 15 years of optimization of the transactions, database schemas and application partitioning that have been built for applications such as ERP or CRM. This raises critical questions about this approach to MDM. Are we really "optimizing" the data or are we going to have to dumb it all down? Some have compared this approach to a return to enterprise-wide IMS and the 3270 terminal. Are we making progress or going backward?

The technical effort required to create an "optimized" single transactional repository would create huge execution challenges for any IT manager and CIO. The design issues associated with trying to find a single set of "universal" transaction structures and a single schema that optimize all critical applications may be impossible. Application architects, database architects and database administrators are gainfully employed today for a very good reason - it is extremely difficult to optimize data, transactions, partitioning and hardware for even a single transactional application. The technical and political challenges of trying to agree on and deploy a single repository would quickly turn young IT managers gray and cause cost overruns for even the most substantial budgets. Not only must the data be consolidated, optimized and restructured, but each application may require massive rewriting to take advantage of this new "optimized" schema. This would result in significant disruption to day-to-day operations. No wonder attempts to actually do this are forecast to take three to five years to reach initial deployment at a cost of $10 to 30 million in services.

Finally, this transactional approach is not optimized for more analytical use. Hence, some IT professionals now believe that utilizing the transactional infrastructure approach will also require a new set of analytical solution architectures to maximize the value of these hubs, particularly for those that must consolidate the data with third-party reference information or other fast-changing external data. Meanwhile, business teams grow increasingly irritated at the lack of IT progress in meeting their business needs. Three-to-five-year projects typically do not help business users reach their goals.

A New Alternative - An Incremental and Pragmatic Approach

Building the Enterprise Master Incrementally

Several vendors are now proposing a different approach to MDM. While each vendor characterizes the architectural approach differently (incremental, solutions-driven, etc.), the concepts associated with these approaches are similar. The approach suggests that the master data hub can be rolled out incrementally, eliminating the necessity for the "big bang" that accompanies the transactional infrastructure approach just described.

Instead of trying to "black hole" the entirety of the data into a single repository, the incremental approach looks at master data from the more pragmatic perspective of what the business needs to meet current project requirements. For example, as a business project such as a customer service portal is specified, the incremental approach allows IT to consolidate only the records and attributes required for the new project. As additional projects arise, the new enterprise hub can identify the appropriate data across the various applications that must be incrementally added to the existing master to extend it to support the new requirements. With this approach, enterprise master data is created over time as needed, instead of utilizing an all-or-nothing approach.

Architecturally, the incremental approach starts with a "reference" information structure that reconciles master data identifiers across the enterprise. Records can be left in their existing systems and are matched into a master via a single master "registry." This minimizes disruption to existing production systems and transactions and allows for composite applications and "single view" analytical solutions to be rapidly deployed. A reference data architecture can evolve to become a hybrid data architecture where master records are unified to create a "best" representation that is synchronized across transactional and analytical systems.

The benefits of this approach to MDM and CDI are quite straightforward.

  • Short time to value. Because new applications can use master records as they are built, IT can quickly support business solution requirements instead of suggesting to the business to wait for a massive physical data consolidation.
  • Flexibility. A hybrid architecture adds the ability to use Web services. Transactions to either update a central data record or to synchronize source systems. This approach offers information architects multiple options for determining the ideal master data and service structure for each project. This approach also provides IT with the option of using the same technology platform to meet both transactional and analytical requirements.
  • Less politics. Because this approach allows incremental deployment, data is only consolidated into the master when necessary. Since all data does not need to be forced into a single physical database, transactional and structural optimizations built for each individual application system are retained, enabling business system owners to retain control over their system and data.
  • Less cost and time. Using the incremental approach, solutions typically can be deployed for an order of magnitude lower cost than solutions taking the first generation transactional infrastructure approach. By not requiring massive re-engineering of each business-critical application in order to deploy a new, composite solution, hubs can be deployed in a manner of months or even weeks.

While the short time to value of the incremental approach makes it extremely attractive, a couple of drawbacks do exist:

  • Completeness. The incremental approach does not get all the data into the hub at once. It does not enable a "universal" transaction engine deployment, so it does not create a global transactional resource.
  • Ultimate scaling. Physical unification of data records ultimately yields the highest level of information scale compared to any reference model approach. However, contemporary distributed computing and load balancing techniques applied to alternate architectures are steadily eliminating the problem of scale.

When deciding which MDM approach to use, organizations should fit the approach to the need. On the one hand, would the resolution of immediate data issues over the next four to nine months benefit the business significantly? Would a long data project cost too much and cause unacceptable disruption to the business? If so, consider taking the incremental MDM approach. On the other hand, if high costs and a three-to-five year time frame are not as important as creating a single, global transactional resource, consider the transactional infrastructure approach. Just make sure you fully consider the political and technical challenges of creating a single database structure and schema.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access