Integrating without the Integration
Data Integration
Information Management Online, January 13, 2005
Last month, we introduced the burgeoning issue of acronyms in the world of data integration. Ever increasingly, our world is being overrun by alphabet soup; that is, we are lost in acronym hell with the lines blurred to near incomprehensibility. Life, for those responsible for ensuring the date quality and interchange between all our corporate applications are concerned, began with a few programmers locked in a room writing custom SQL and file extracts to share limited amounts of data. Then we moved on to ETL, EAI, EII and EIEIO...
As we posited last month, the real issue is not the fact that data needs to be shared. The issue comes from the nature of the current business environment. Conglomerates have taken over the family-run, corner markets. The era when a large, successful business could be run on a single, entry-level accounting package are long gone. The times when executives could gather all the information necessary from a few canned reports are long gone. The business world today revolves around many sophisticated and specialized applications, such as general ledger, accounts payable, human resources, sales automation, customer relationship management and supply chain automation. Toss in a few homegrown or customized applications to handle inventory management and other ancillary activities, robust and dynamic reporting, analysis (OLAP) and dashboards (scorecards) and you have a complete picture of today's business environment.
Obviously, custom-developed SQL extracts will never suffice. The integration functions must be flexible and easily maintained in order to accommodate the rapidly changing nature of business interaction. Extract, transform and load (ETL) products were the first on the market to accommodate the need for business applications sharing data. It was ETL that we discussed last month. This month, we will focus on the "new kid on the block" - enterprise information integration (EII).
Advertisement
The need to provide senior executives with real-time access to information from throughout the organization results in just about every company needing to take a step back to evaluate their integration strategies. Enterprise information integration technologies allow for this real-time access by simplifying and managing the access to disparate corporate information. In a move completely opposite from the way that ETL and the other integration methods work, EII identifies related data from multiple back-end sources, provides a facility for the user to issue a single query that retrieves the pertinent information from the different applications, and consolidates and aggregates the data in real time for presentation as though the information was all contained in a single data repository. By implementing this type of integration technology, any possible variety of information contained in the company's data coffers becomes available for use by the users at greatly enhanced access times. Further, the information can be presented in ways most useful to the end user. The organization's user types may be defined to the EII server, which then presents the data to the user in a default view that is most relevant. For example, the VP of sales may retrieve the sales reports data from the EII implementation. This data is the aggregated summary for the entire organization and may present total sales figures, plus information about the top five products sold, plus the top five issues (such as lapsed customers or late payments). Once within the presentation of the data, the VP of sales could then drill down to specific regions, products, salespersons, etc. The fact that the EII query application is gathering the data from the ERP, the inventory management, the SCA and the CRM applications is not known to the VP, nor should he be concerned with the data's ultimate location. The application provides a layer of abstraction between the client's query tool and the various databases and source systems. As far as the user, and his query mechanism, is concerned, he is looking at a dashboard of information from his enterprise data warehouse or sales data mart.
It sounds as though the EII approach is the panacea of data integration. So, why should anyone ever want to enter into the world of data warehouses and traditional data integration ever again? Well, the EII technology is significantly different from ETL and EAI-based solutions. EII performs many of the same data conversion and aggregation tasks as the other forms of integration. The key differences are that extracts files are not used as sources, all of the data processing is done in the EII server and the final target for the integrated data is the user's presentation layer, not a data repository. Well, there's the rub. In order to accomplish this, the EII technology requires highly sophisticated data caching, data indexing and distributed query optimization algorithms. Just as the other integration methods do not allow for real-time integration and presentation to the user, they do not require these sophisticated mechanisms which do result in the EII implementation being a bit more intellectually challenging than ETL and/or EAI.
EII is still in its infancy. There are only a limited number of new players in the marketplace that are delivering the EII technology; however, the larger integration vendors are updating their batch-oriented ETL products to allow for some level of virtual integration and user presentation in order to enter the EII fray. All the current EII vendors are relying on open standards - XML, SQL and Web services - upon which to base their product offerings. Their key differentiators are their patented caching, indexing and query optimization algorithms. Since the vendors are all supporting open standards on the front and back ends, they are the logical choice for enterprises to rely upon for many of their future data integration needs. This is especially true in light of the "legacy ETL" vendors offering EII-like capabilities. It is only a matter of time before batch ETL processes are replaced, in large part, by EII applications.
Over the past few months, we have covered opposite ends of the integration spectrum. We have discussed integration's infancy, ETL, to its current methods, EII. In the past, we have also discussed the needs that businesses are showing for hierarchy, reference data and unstructured data management.
Page 1 of 2.
