In my previous column, we started to look at the topic of master data management and customer data integration, and how those tasks are related to each other as well as to other historical attempts at enterprise knowledge integration. As part of the integration process, this month we look at a case study involving the exchange of data using what are thought to be commonly defined data standards. This case study exposes a problem when a data standard is in place and what happens when individual participants adjust their adherence to the standard.

This topic surfaces frequently, and it relates to the use of agreed-to data set formats for data interchange whose use somehow diverges from their original intent. The changes occur subtly over time, and in potentially multiple places; therefore, until an analyst actually sits down to review the different uses, the variants would not even be noticed. Sometimes the differences are in content and sometimes they are in form, but at some point, especially when data is being exchanged, the variances will result in the inability to synchronize data values between information exchange partners.

In this case study, we were consulting with a client who sends and receives data with a number of information exchange partners. There is an agreement in place to include routing codes in exchanged messages that help redirect messages and requests to the appropriate location within an exchange partner's local environment. When partner A sends a request to partner B, A attaches a routing code related to the originator of the request within the message. B's reply to A will include the same routing code, which A then uses for redirection of the reply to the originating requestor.

Each partner is responsible for its own scheme for assigning routing codes as well as its own process for message redirection. In addition, all the partners post some version of their routing codes to a centralized repository so that anyone originating a message can look up the routing code, append it into a new message and believe that it will be routed to the proper location at the destination. However, there is some (limited) enterprise-wide guidance describing a general scheme for data formatting, and all of the exchange partners believe they conform to this guidance.

Here is where the subtle problems occur. Despite their best beliefs, a number of the exchange partners have, in their own ways, slightly modified the way that the routing codes are formed and how they are mapped internally. Those who have done this have good reasons, but they have adapted variance into the process without obtaining agreement from the other partners. There are a few results as follows:

  • Those partners that have made changes have adapted the same physical components of the code for different meanings. One partner may use a part of the code to indicate routing to a centralized office, while another uses that same part of the code (and sometimes, the same mapped values) to indicate routing to a financial accounting service.
  • Some of the partners have added extra bytes to the structure, allowing for additional information to be appended. Yet those that have not extended their data structures are unable to capture the appended values, which means that the full values cannot be incorporated in any response. When this happens, the messages are either improperly routed or are dropped altogether.

As these variations have crept into the environment, the knowledge-workers that participate in the workflows have begun to notice the existence of problems, but they have had difficulty determining the source of the problem. The reason is that even though the adjustments that each participant has made only slightly alter their conformance to the standard, in combination across the extended enterprise, a number of these variations clash with each other. As a result of the limited enterprise-wide guidance, conflicts emerge from the loose adherence to the standard. This, in turn, reduces the efficiency of the interacting systems.
For example, while partner X uses a specific code (e.g., "000") to indicate routing to a central processing office, partner Y uses the same code for routing to an EFT/EDI processor. Therefore, when partner X wants to send a message to partner Y's central processing center, X's use of the "000" code in its message results in Y routing the message to a completely unintended location.

Even though this case study looks specifically at an exchange network, the lesson learned can apply directly to data integration across an enterprise. There are always expectations of conformance to some standards, but often in reality there is no central oversight of these standards. They are rarely properly documented, and the IT staff within each line of business does not feel constrained by compliance to a standard. This complicates data aggregation and integration, especially for data warehousing and business intelligence applications.

The solution to this divergence from what is believed to be a standard must facilitate the cooperation of the participants along two dimensions: precise definition and adherence to the standard. Because of the limited guidance, there were enough loopholes in the standard to allow enough variation until conflicts became inevitable. By having more precise (or, perhaps, stricter) definitions and guidelines, the opportunity for introducing variation is reduced; and, consequently, so is the possibility of value clashes. Second, it is to each participant's benefit to completely conform to the standard. When everyone plays by the rules and knows that everyone else is playing by the rules, data expectations may be set properly. The result is a more streamlined ability to exchange and integrate information across the enterprise.   

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access