This is part of a series of columns on business intelligence (BI) and data warehousing (DW) trends for 2008.
How’s this for irony? A lot of integration projects create more silos!
This happens because companies implement integration technology and products on a project-focused basis. The integration projects concentrate on specific applications and processes, in other words they integrate silos. Each tactical integration task is completed, and eventually the company discovers that they need to integrate the silos created with their integration projects.
The bad news is that many companies have already created these integration silos. The business groups that invested in these integration projects are now back to using spreadsheets, or data shadows systems, to reconcile the conflicting numbers coming out of these silos.
The good news is that many IT and business groups are educated consumers. They have seen the result of their myopic integration efforts and understand they need to change their ways.
The Integration Sea Change
What should you do if you find yourself in this situation? Step back from the tunnel vision of tactical data integration projects and design an overall integration architecture. After you have this overall architectural blueprint you can then design your individual projects, fitting them into this overall blueprint. Just like you hand your house blueprint to various contractors so that everything fits together, you need your data integration blueprint to get things to work together too. This blueprint encompasses architectures for data, technology, product and information (business data transformed).
Enterprise data management (EDM) is the blueprint (or holistic approach in consultant lingo) that people should be striving for. Integration efforts, be they enabled using enterprise application integration (EAI), service oriented architecture (SOA), enterprise information integration (EII) or extract, transform and load (ETL) technologies, all involve integrating data.
Data integration processes involve mapping one or more data sources to a data target and transforming the data along the way. Regardless of whether you are using messaging, services or batch-driven technologies you are performing the same processes, just the transport is different.
Rather than taking a common approach, most integration projects start with an entirely different set of technologies or products. However, these products and technologies overlap and, more importantly, the data they integrate overlaps. The result is silos and business people manually reconciling the data they just spent a lot of money and effort integrating.
Fortunately, if your company is in this situation you are not alone (misery loves company) and great options have emerged in the market. First, there are best practices to design your data integration framework (DIF) or blueprint. Second, there are proven program and project approaches that incrementally build an EDM and migrate from your silos.
Finally, some of the top ETL vendors have recognized the need for a comprehensive approach and have transformed their ETL products to data integration suites. These suites have expanded beyond batch-oriented ETL to include EAI, EII and SOA. In addition, it is becoming increasingly common for these suites to offer data profiling and data quality functions. No longer are you forced to buy separate best-of-breed products supporting niche technologies. You can find products that will support your DIF.
The market leaders, according to Gartner, Inc. are Informatica and IBM.1 Both companies have expanded their data integration capabilities through acquisitions; most notably IBM’s Information Server obtained a significant portion of its functionality from its acquisition of Ascential Software.
Be the first to comment on this post using the section below.