Next generation data management tools must function as an "integration technology" that supports and services data integration, data warehousing and, ultimately, e-business applications and services. ETL tools have traditionally been used to collect and store strategic enterprise analytics and decision support data. Now enterprise application integration (EAI) tools have emerged to fill the gap left by data warehousing technology and respond to the real- time needs of the Internet and other applications. EAI is providing the real- time data link and glue needed to tie the enterprise resource planning (ERP) systems which automate the supply chain and the internal operations of the enterprise, the customer relationship management (CRM) systems that automate the selling side of the enterprise and every-thing else in between. Unfortunately, there is still a gap because EAI tools do not provide the extract and loading capability of the ETL tools.

A third solution has also emerged with the data query management (DQM) tools that bypass the data warehousing architecture and provide real-time data access and integration in heterogeneous DBMS/platform environments. This may be ideal for limited volumes and transactions that have many applications, especially when merged with powerful OLAP/reporting tools, but this is not the complete answer. GartnerGroup predicts a technology transition in the market merging or replacing the ETL, EAI and DQM tools into flexible, scalable, intelligent information logistics networks (ILNs) that route data to and from information- craving entities based on prescribed business rules. This transition will not take place overnight, but it will most likely grow from vendors already in this marketplace.

ETL vendors are ensuring that their ETL tools interface with process integration and EAI tools in order to position their ETL tools as the means for getting, sharing and integrating structured and unstructured data from one system to another. The market for these tools is changing ­ as is the competition ­ with the introduction of e-business integration broker products and enterprise application integration tools which encompass the full range of data transformation capabilities needed for integration and management of business processes and transactions across ERP and CRM systems.

Initially, ETL tools were created to deal with the mismatch of operational storage of information and the need to use that data with other applications, especially data warehouses. When there was impedance between the source and target data structures, the ETL tools automated the arduous coding task of mingling data and created a combined model that was acceptable across numerous tools. The next generation ILNs must move beyond the basic process of mingling data from disparate sources and be able to extract, transform and load both structured and unstructured data. "If ETL tools remain the same, they will quickly become a commodity, and the functionality will be consumed into other technologies," says Mike Wipperfeld of Ardent Software. Therefore, ETL must evolve into integration technology that solves issues at the data level and beyond. Some ETL vendors are making their tools critical to the process of sharing data from data warehousing to business to business (B2B). They foresee that eventually meta data will be XML- based because, as an interchange format, it offers a great deal of flexibility.

ILN, in the broader sense of integration technology, must afford and require the ability to move from batch to real time and structured to unstructured data, and manage and share technical to external industry data. As you reach the highest level of data integration, you approach e-business. At that level, the sharing of information across systems is at its peak. The fourth dimension of integration technology is performance and scalability. ILNs will take advantage of underlying software and hardware, and leverage parallel processing from source and target.

Many organizations are putting their data warehouses on the Web in order to understand the relationships between consumers' buying decisions and trends. The new breed of data management tools must provide access to real-time, Web- based applications to connect to the real-time databases that contain: specific client profiles; logistic information about the anticipated delivery cycles and schedules; production cycles for clients and sales to insure that demands can be met; and inventory levels for clients and retailers. The ILN must also collect the interactions taking place and send them securely to the data warehouse for analysis and actions. With a closed-loop data warehouse, passing actions must be passed back to the delivery channel Web-based interface appropriate to the request and fit the profile of the client, retailer or internal representative that launched the request. These real- time, Web-based interaction challenges will be met with the new generation of data integration tools and the marriage of ETL, EAI and DQM ­ yielding ILNs.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access