Key mapping, the linking of unique identifiers from multiple systems, is a widely used technique to address data integration challenges brought about by having common information scattered across multiple applications. The typical enterprise uses key mapping today for data integration because databases employ local unique identifiers as the most common navigation path to a piece of data and because most enterprises have the unfortunate need to store identical, similar or related facts in many places. Some optimists claim that this scattering of data will subside as monolithic enterprise applications deliver required business functionality. However, most realists understand that the scattering may subside but will never disappear. Others fantasize that the panacea of master data management (MDM) will bring us to the more enlightened state of one fact in one place. Great stuff, but during this slow and painful journey to paradise, we mere mortals must continue to rely on key mapping to rationalize native identifiers of common legacy data.
The key mapping opportunity exists for many data subjects, including customer, product, person, place, machine, state and even simple domains such as activity status and unit of measure. Regardless of subject, it turns out to be the very same logical problem. We need to know that this unique identifier over here corresponds to that unique identifier over there to meet the everyday requirement of integrating this common data appearing in multiple applications.
We all understand the underlying business need to consolidate, integrate or synchronize common data from multiple applications. After all, this is what IT does. Overall, integration could be the most common and expensive requirement in all of IT. I am sure that examples of data integration opportunities abound in every organization and need not be listed here. It probably would not be too risky to assert that 80 percent of all IT pain comes from this business requirement alone. At the same time, data integration may be what provides 80 percent of the business value, and it may also be the source of 80 percent of business dissatisfaction with IT. Sure, its painful, but we gotta have it.
However, integration is not really the problem addressed here. Sure, it is an opportunity, but one that has always existed, and IT continues to provide data consolidation, integration and synchronization as best it can. Key mapping design is not necessarily the problem, either. It is pretty straightforward - certainly not rocket science - although there is often some complexity arising from semantic dissonance and from schema reconciliation required for subtly different implementations of a common concept, such as physical address. There are many ways to solve key mapping, and most solution architects have found a way to address it one way or another.
The much larger business problem is the more fundamental issue that IT is wasting important resources (time, money, talent) solving this same challenge repeatedly in different ways and with limited scope. Not surprisingly, this results in duplicated effort, unnecessary complexity, inaccessible translations, redundant and conflicting mappings and maintenance spread across multiple areas of expertise. The sucking sound of important resources going down the drain should be deafening, but no one hears it because the key mapping design effort is repeated quietly within every project that has integration requirements (ah, wouldnt that be every project ever?). IT is applying small amounts of time, money and talent on key mapping design so frequently that it adds up to large amounts of time, money and talent. Now thats a problem.
To address this ubiquitous key mapping opportunity, the typical enterprise has the full gamut of solutions, developed independently over the years, each with its scope limited to a particular type of data. Each implementation may be dedicated to a particular data flow, application, database, subject area, development platform or integration technology. The scope is always limited. Consequently, multiple independent solutions exist for essentially the same problem, sometimes right next to each other at the same data integration point.
Perhaps the key mapping is hard coded in a data interface using if-then-else or extract, transform and load (ETL) transformation logic (IF source.status_key = A, THEN target.status_ID = 2). Or maybe a column has been added to a local database table to store another systems key (SAP_VENDOR_ID). Another approach is to dedicate a local table to a particular mapping (VENDOR_XREF) or a generic table for all local mappings (XREF table with a XREF_TYPE column). Perhaps mappings are managed within integration middleware. Each of these solutions bites off a larger piece handling a broader scope, but there is the potential for overlap of solutions and key mapping discrepancies when many solutions coexist.
Global Keys Solution
Future-thinking architects are gravitating to a universal key mapping solution known as global keys, designed generically for multiple data subjects, implemented once, wrapped in Web services and utilized broadly across the enterprise.
The premise of the global keys approach is that centralizing key mapping as metadata and isolating it from any particular application is the most effective way to increase its value as a sharable enterprise information asset. We can think of global keys as a master repository of keys, perhaps implemented as an extension to an existing metadata repository, managing information about technology components such as applications, databases, tables, columns, servers, workstations, etc. When we view key mappings as metadata, we realize that they should not be owned by a particular application, not even MDM. Cloistering key mappings within an application diminishes their value by limiting exposure, access and reuse across the enterprise. This is happening everywhere and every day.
The global keys architecture isolates key mappings from data content. Global keys contain only native keys and their associations. We know that these keys are immutable (lets hope so - keys that change are their own special problem), and we suspect that their associations are stable, but the data content itself is likely to be more volatile. In global keys, we capture the keys and support the management of their associations. Where and how to manage the actual data content can be addressed independently on a case-by-case basis. In the system of record? Distributed across multiple applications? For each subject area, this decision will depend on many factors, and it will likely be a volatile mix of these data maintenance platforms for years to come. However, key mapping does not need to be addressed independently and does not need to be a volatile mix of platforms. A safe and sensible decision can be made to deploy the global keys architecture centrally and stabilize the key mapping aspect to build an enduring and valuable enterprise asset.
The global keys architecture also isolates the management of key mappings for a data subject from the actual match determination made with human intervention or using domain-specific tools such as those for customer or product data management or for MDM. In a sense, we pull the management of persistent mappings out of MDM and put that management into a set of services focused entirely on registering keys and updating their associations. Decoupling the data mapping determination from the results enables various mapping tools and techniques to handle the unique matching challenges of their special domain while using a common key mapping data resource. The consolidated global keys data resource can then be exposed broadly (on the enterprise service bus, perhaps) and accessed widely under a single security framework. At last, in lieu of one fact in one place, we can deliver and manage one key fact in one place.
Once established, the global keys asset can be consumed broadly for integration opportunities such as application integration, data consolidation and data integration commonly present in any enterprise. Each of these types of integration has a need for key translation.
Eliminate Cost Redundancy
Any time multiple solutions or patterns emerge for the same problem, as they have for key mapping, cost-reduction opportunities exist. When you think of all the assets that need to be managed to support any solution lifecycle, the costs begin to add up. Outlays for technology, licensing, design, documentation, application code, training, tools and upgrades are required for every type of implementation. We know that cost containment has been a top issue for IT over the years, yet we continue to reinvent the wheel time and time again. It follows that consolidating the various key mapping solutions under a unified global keys architecture will help reduce redundant costs and cut IT spending for the enterprise.
Single Scalable Solution
Providing a single scalable solution for a common opportunity is a worthy goal for any information or solution architect. Designing once and utilizing many times delivers benefits beyond providing the enterprise with an efficient utilization of design resources and typical cost savings. The narrow functional scope and broad applicability of global keys promotes a flexible, extensible design yielding a stable and durable implementation and providing value for the long term. The global keys architecture provides a focused, flexible, simple solution to one of the most widespread integration needs in all of IT - key mapping to support application integration, data consolidation and data integration across all data subjects.
One Fact in One Place
One key fact in one place wrapped in a comprehensive set of services provides a universal platform to be accessed widely as a truly reusable information resource. With a global keys architecture, the entire technical enterprise knows where to go to tap into this valuable knowledge asset. Any new data consolidation opportunity, such as a new aggregation or cross-application integration, may easily leverage existing key mappings. Once the global keys infrastructure is established, a new source or new domain can be defined to immediately register and persist a necessary cross-application key mapping with only a small investment in Web service calls or messages. Global keys provides a crucial component for successful data integration.
A global keys implementation maps application data stores such as tables to global entities. The existence of global entities suggests an enterprise logical data model that defines the hub of the global keys architecture. The first step of a global keys implementation is the development of a global or enterprise data model defining the common global entities having keys to which local keys are mapped. Of course, we know that an enterprise logical data model exists for every enterprise (including yours). In some enterprises, it is explicit; in others, implicit. As in most endeavors, explicit outperforms implicit. I would be preaching to the choir for those who understand this and wasting my breath on those who do not.
Because global keys are essentially metadata, decisions are required around integration with existing metadata. Along with keys, the global keys architecture describes data repositories, data stores and logical entities that may already be managed as metadata. Perhaps the global keys metadata of keys can simply extend the existing metadata repository. Or perhaps a new home for global keys metadata is established with metadata itself being the first domain to map. Scary.
Establishing administrative Web services or user interfaces (or both) to manage definitional metadata is a prerequisite to capturing keys. Once again, being explicit about roles, processes, approvals and the governance of global keys and its associated metadata is conducive to a successful implementation.
Comprehensive services enabling the registration and assignment of keys provide the interfaces required to build the global keys information resource. Transactional services are packaged to support bulk-loading requirements. Exposing these services using precise and unambiguous descriptions, training, consultation and ongoing support also fosters success.
Implementations of small independent domains can help build momentum and gain repeat customers when successful. Starting with relatively static mappings that are commonly referenced helps deliver confidence in the global keys platform. Building key mapping assets that have greater complexity, such as one-to-many mappings (i.e., one local table to many global entities), can be implemented with confidence over time.
Deploying a global keys architecture to address the commonplace key mapping opportunity as a unified, extensible utility is a cost-effective solution that can deliver a popular enterprise service that is narrow in scope, broad in utility, simple in design, quick to implement and centrally supported.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access