Data integration suffers from an image problem. It has become synonymous with extract, transform and load. Likewise, ETL has been regarded as a data warehousing technology. Both of these viewpoints fail to reflect current capabilities, and they greatly inhibit enterprises in their attempt to integrate data to provide the information their business needs. Because of this short-sightedness, companies have lost opportunities to harness information as a corporate asset. It increases the cost of integrating data, encourages the creation of data silos and forces businesspeople to spend an inordinate amount of time filling the information gaps themselves through data shadow systems or reconciling data.
More than Simply ETL Tasks
The basic tasks required in data integration are to gather data, transform it and put it into its target location. If that sounds like ETL, that’s because that’s exactly what it is (see Figure 1). ETL tools have automated these tasks and empowered developers with a toolkit beyond what they could have easily hand coded themselves. ETL tools, for example, include prebuilt transformations from the elementary tasks of converting data and performing lookups to the more complex processes of change data capture and slowly changing dimensions. These prebuilt transformations greatly enhance developer productivity and improve the consistency of results. Data integration tools offer many significant processes and technologies that extend beyond the basic ETL tasks (see Figure 2). These extensions are necessary to turn data into comprehensive, consistent, clean and current information. The extended processes include data profiling, data quality and operational processing. This creates the ability to determine the state of the source systems, perform cleansing, ensure consistency and manage all the processing, including error handling and performance monitoring.
Data integration suites have expanded to incorporate enterprise application integration, enterprise information integration and service-oriented architecture coupled with ETL to offer data integration in batch, interoperating with applications, or in real time from business intelligence applications.
More than Batch Processes
There are many integration initiatives in an enterprise. With data integration being associated with batch-driven ETL processes that load a data warehouse (DW), integration initiatives that did not involve a DW sought out other technologies. These technologies included EAI, EII and SOA. Although each of these technologies has fundamental applications, the reality is that organizations had to reinvent the wheel for every data integration task. The results have been integration silos built with different technologies, producing inconsistent business information and generally with data integration built as an afterthought.
The good news is that data integration vendors that now combine all of the above technologies into data integration suites have emerged from the ETL ranks. These suites enable an enterprise to integrate data in one consistent manner, yet deploy using whatever transport technology (i.e., ETL, EAI, EII or SOA) is appropriate.
More than Data Warehousing
With the emergence of the more powerful suites, data integration has moved beyond data warehousing to include other integration initiatives in an enterprise, such as:
-
Data migration,
-
Application consolidation,
-
Operational and real-time BI and
-
Master data management, customer data integration and product information management.
Companies often undertake data migration or application consolidation projects because of mergers and acquisition or because they need to streamline applications. In the past, these projects were seen as one-offs and typically hand coded. As systems integrators became proficient in ETL tools from DW projects, they realized that they would be much more productive at data migrations and application consolidation projects if they used these same data integration tools. Even though they are one-offs, data integration tools enabled the SIs to reuse code, leverage prebuilt transformations, better manage processes and produce documentation without a laborious manual effort. In addition, they did not have to deploy a cadre of coding gurus but could leverage the data integration developers they already employed.
Several market forces have converged to produce the perfect storm, enabling operational or real-time BI with the same data integration and BI tools as used in DW projects. These forces include enterprise applications built on relational databases and data integration tools no longer bound to batch ETL constraints. In addition, with the major enterprise application vendors also offering data integration and BI tools, these vendors are bundling this convergence is more consistent, comprehensive and current information (business benefit) with the same data integration and BI infrastructure (IT benefit).
MDM, CDI and PIM all deal with conforming and maintaining master data or reference data for data subjects, such as customers and products. The initial wave of technology solutions bundled a set of tools and applications that were business process or industry specific. What got lost in many of the initial implementations was that these applications relied heavily on data integration and that it made sense to leverage a company’s existing data integration platform to create MDM, CDI and PIM solutions.
