Virtual Versus Physical Data Integration: How to Decide

InfoManagement Direct, January 16, 2009

“If all you have is a hammer, everything looks like a nail.” Barnard Baruch’s quote resonates in IT organizations that often find themselves solving new problems with their old reliable tools even when more appropriate new tools may exist.

This truism especially applies when it comes to choosing data integration (DI) methods. Physical data consolidation - the combining of related data into a common physical store using ETL tools - is like a familiar, reliable hammer. Data virtualization is like a screwdriver. Which will be the most useful to solve a problem? Like their toolbox counterparts, both are indispensable for solving different DI problems. The challenge to IT comes in deciding early in the design cycle which tool best fits the job.

A Key Capability in Every DI Portfolio

For enterprises around the globe, ETL has been the traditional “go to” tool for integrating data across disparate sources. However, data virtualization has moved from an interesting new approach to a standard DI method along with physical data consolidation and data synchronization (see Figure 1). Data virtualization brings together (federates) data from multiple, disparate sources - anywhere across the extended enterprise both inside and outside the firewall - into unified, logical, virtualized views or data stores for consumption by nearly every front-end business solution including portals, reports, applications and others. As a project-oriented DI middleware, data virtualization is often referred to as virtual data federation, high-performance query or EII. As enterprise architecture, it is frequently described as a virtualized data layer, an information grid, an information fabric or as data services in SOA environments.

Industry analyst firm Gartner, in its June 2008 report, “Survey on Data Integration Practices Shows Move Toward Strategic Initiatives,” shows that more than 50 percent of organizations surveyed say they are creating virtual integrated views of data from disparate databases via data federation techniques. Industry analyst firm Forrester Research, in its October 2008 report, “Securing Next-Generation Information Architectures,” sums up this trend: “Next–generation information architectures such as data federation and information services are gaining increased adoption.”

Figure 1: Data Virtualization within a Data Integration Portfolio

(For a larger version of Figure 1 see PDF below.)

Making the Right DI Decision

Nearly every new solution that IT builds leverages data from existing sources and therefore can benefit from data integration. Data virtualization is a natural fit for many use cases; physical data consolidation is the right answer for others. Sometimes, the best solution is a combination or hybrid of the two.

Recognizing the importance of this decision, DI industry analysts and software vendors, working in concert with data architects and integration specialists, have published useful DI decision-making guidance. Among the offerings available, there are two distinct decision-making approaches: integration pattern matching and integration factor analysis. Both may be applied to individual projects using insight readily available in typical project justifications, higher-level designs and detailed specifications.

Integration Pattern Matching

Figure 1: Data Virtualization within a Data Integration Portfolio

Figure 2: Relative Weighting for Decision Factors

Page 1 of 4.