Data Integration: Still a Barrier for Most Organizations

Information Management Magazine, April 2006

Colin White

Data integration was hot in 2005, and the intense interest in this topic continues in 2006 as companies struggle to integrate their ever-growing mountain of data. A TDWI study on data integration last November found that 69 percent of companies considered data integration issues to be a high or very high barrier to new application development.¹ To solve this problem, companies are increasing their spending on data integration products.

In this article, I review key data integration trends and look at how companies are exploiting new and evolving data integration approaches to break down barriers to data integration in their organizations. The four main trends I address are an enterprise approach to data integration, integration competency centers, right-time data integration and master data management (MDM).

Enterprise Data Integration

Although data integration problems are widespread in companies, much of the focus by IT organizations to date has been on integrating structured data in support of data warehousing and business intelligence (BI) applications. The BI market has seen tremendous growth over recent years and, for many companies, BI has become mission-critical because of the important role it plays in the decision-making process. This reliance will increase as companies move toward using BI, not only for strategic and tactical decision-making, but also for driving daily and intraday business operations.

The use of data warehousing and BI has led to a much better understanding of how business data flows through the business and how it is used to make decisions. This is especially true for legacy system data, which is often poorly documented. This understanding is helping organizations deploy other data integration projects that may not be directly related to BI. The result is that more organizations are now viewing data integration as an enterprise-level problem, not just an issue to be solved when building data warehousing and BI applications. The TDWI study showed that 18 percent of companies have developed an enterprise-level architecture for supporting data integration in their organizations.

Figure 1 shows the data integration architecture I use when discussing an enterprise-wide approach for managing and deploying data integration projects. As you can see from the diagram, the architecture addresses four aspects of data integration: applications, techniques, technologies and management.

Figure 1: Data Integration Architecture (Source: BI Research)

A data integration architecture defines the underpinnings of an enterprise-wide data integration solution. It provides a starting point for each data integration project. The actual applications and technologies (and thus products) selected will be determined by the business and technical requirements of each project, but the data integration architecture provides the basis from which these selection decisions can be made.

Integration Competency Centers

As companies move toward an enterprise approach to data integration, they need to reuse and share data integration expertise and resources. This can be achieved by creating a data integration competency center. These competency centers come in many shapes and sizes. Some are funded directly from the IT budget, whereas others act like an outsourcer and charge projects for services. Some centers focus on defining a technical architecture and a preferred product set, while others also provide data integration expertise and development resources to projects.

Some competency centers evolve from an organization's data warehouse development group. In other cases, a company may merge several areas of integration expertise into a single integration competency center (ICC). An example here is the merging of application integration and data integration groups of expertise.

The use of BI to drive intraday business operations requires that BI systems become tightly integrated with operational systems. These operational BI projects often blur the dividing line between what is operational processing and what is decision processing. If the two types of processing are handled by different groups, then this can lead to ownership and technology disagreements. The solution is to pool the expertise and resources in a single ICC.

Setting up an ICC requires strong management backing and enforcement, otherwise it will fail. Gartner Research notes, "The top-performing one-third of ICCs will save an average of 30 percent in data interface development time and costs and 20 percent in maintenance costs, and achieve 25 percent reuse of integration component during 2004 through 2007. The remaining two-thirds of ICCs will fall short of those benefits because of insufficient sponsorship and other organizational execution problems."^2,3

Right-Time Data Integration

As mentioned previously, one of the biggest growth areas in BI lies in its use for driving daily and intraday business decisions and operations. Operational BI often requires access to more current data than that provided by a traditional data warehousing and BI system. This requirement eventually leads to a debate about how to supply real-time data to operational BI applications.

Real time is a politically sensitive term from an IT perspective, and it is a misleading way to approach operational BI because it suggests that BI applications can access, process and deliver information in real time. For all intents and purposes, real-time delivery is not possible because there will inevitably be latency in the information delivered to the business user compared with the live data being managed by operational business applications. This position depends somewhat on what you consider to be real time and what you consider to be BI.

To many people, real time is the ability to receive information about an event occurring within a fraction of a second. Operational business applications, for example, operate in real time because they instantly supply data to business users. In actuality, these applications may take several seconds to respond, but typically this response is timely enough that users don't get impatient and start hitting the enter key on their keyboards!

A reporting application that provides data (order information, for example) about customers to a support representative in real time could be considered to be a real-time BI application. To be realistic, however, the purpose of operational BI is to analyze data and convert it into useful information that can be used to make rapid decisions. This processing involves gaining access to the required data, analyzing it and delivering the results to the business user. The user needs to analyze the information further, make a decision as to whether any action is required and then take any required action. This complete process cannot happen in real time. The latency in the process will depend on business requirements and the ability of the BI technology to support those requirements.

Page 1 of 2.