This article is excerpted from Chapter 2 of Proven Portals (Addison Wesley, 2003) by Dan Sullivan.
The hallmark of a portal is integration. Portals provide single points of access to applications across the enterprise. These applications all function within the integrated framework of the portal but to varying degrees. At one end of the spectrum, we have shallow integration, which brings applications together at the interface level. For example, utility applications, such as stock quotes, airline schedules and currency converters, are all accessible from a single location in the portal and are, therefore, in a limited sense integrated within the portal framework. The applications themselves do not exchange data or in any way depend on each other they simply coexist in close proximity within the user interface. The more interesting examples of application integration utilize a deeper level of interoperation and will be the focus of this article. The challenges, and benefits, of application integration occur behind the scenes of the portal interface where we find architecture and process issues that are as old as information technology (IT) itself.
The business driver behind integration is the fact that multiple IT systems are often required to complete a single business operation. For example, accepting an order online from a customer could require a series of steps that use or update information in inventory, customer, billing and fulfillment databases. The ease with which the online order is filled depends upon how these systems were designed, how the enterprise has grown and how business processes have changed. In the ideal world, these distinct systems were designed with the current business processes in mind and with an eye to integration with other applications. This is rarely the case. Changing business models, mergers and other competitive pressures are constantly forcing organizations to rethink their operations. Portals are one of the arrows in the IT quiver to help implement business strategies, and application integration is a key facet of portals.
Application integration is not a new IT process, but the methods for conducting enterprise application integration (EAI) are becoming better understood. In the past, point-to-point integration was often used to exchange data between applications. For example, if an underwriting system needed data from a claims processing system, a custom data exchange would be built. If a billing system needed information from the same claims processing system, then another exchange would be built and so on for any other application that needed data from the source system. The limitations of this architecture are obvious. The number of exchanges grows with each new point of integration, and each of these needs to be maintained. A single change in a source system can cause ripple effect changes to any or all of the exchanges. A better approach is to standardize on a middleware model exchange in which applications serve data to a middleware broker or mechanism that is responsible for getting the data to its ultimate destination in the appropriate format.
Middleware for enterprise information exchange is essential for many portal applications. The promise of portals, a single point of access to multiple business processes and applications, depends upon the system's ability to complete logical business processes, regardless of the underlying application architecture. For example, a customer making an order online does not need to be aware of inventory, fulfillment and billing systems. To isolate the users from irrelevant (from their perspective) implementation details and to ensure that the integrity of business processes is respected, portal applications depend upon enterprise application integration techniques. EAI systems provide two core functions: the ability to create messages that are meaningful to both source and target systems and the ability to guarantee the delivery of messages between systems.
One can design the middleware for application integration using two different models, the hub-and-spoke and messaging models.
The hub-and-spoke model provides a central routing server, the hub, which accepts messages from clients, the spokes, and routes them to the destination spoke. The advantage of this model is that senders and receivers do not need to know details of the other implementation. If one spoke, or client, changes, only the hub needs to accommodate the change, not the other client applications.
Messaging systems utilize a bus model rather than centralized server model of the hub-and-spoke model. When applications exchange data, the source system creates a message with the relevant data and submits it to a queue, which guarantees its delivery. The main elements of a messaging system are:
- Adapters or gateways,
- A meta data directory,
- A message queue,
- An audit trail database,
- A routing service,
- Transformation and mapping services.
Adapters are application-specific connectors that link applications to the message broker. The meta data directory maintains information about systems using messaging services and support message reformatting and delivery. The message queue is the heart of the messaging system and is responsible for accepting messages and ensuring they are delivered. The audit trail database records messages sent through the system for later reporting and analysis. The routing service supports additional functionality above basic queuing, such as publish and subscribe services. Transformation and mapping services reformat messages to ensure they are syntactically acceptable to the target system.
Some messaging systems, such as IBM's MQ Series, are applications themselves while others, such as the Java Message Service API, are service components upon which other applications are built.
In addition to variations in architecture, EAI systems are distinguished by a number of other characteristics.
Some EAI applications are loosely coupled while others are tightly coupled. Loosely coupled systems tend to operate fairly independently and exchange relatively small amounts of information. Tightly coupled systems depend more heavily upon each other and share more data than loosely coupled systems. The more tightly coupled the systems, the more constrained the business process that dictates the exchange. Loosely coupled systems are more easily integrated. From the perspective of a portal user, the loosely versus tightly coupled exchange is transparent; it is the portal application designer that must understand and address the implications.
Another distinguishing characteristic of these applications is whether they are synchronous or asynchronous. In the former case, the source system must wait for a transaction to complete on the target system before proceeding. This is a potential bottleneck and can constrain the source system unnecessarily. Asynchronous messaging, on the other hand, does not require the source system to wait for the target system to complete an operation. This type of operation allows for greater flexibility in design and eliminates a potential slowdown in source system processing. For example, a customer placing an order online will need to wait while credit is verified; therefore, the synchronous model is appropriate. Sending order information to a billing system can be done asynchronously because a response from the billing system is not needed immediately.
EAI is an underlying technology that allows us to implement entire business processes within a portal; it is the glue that holds complex business processes together. Building portal applications within an EAI framework allows for greater flexibility in business processes and minimizes the impact of changes in one area of business (e.g., a change in a billing system) on other dependent areas. Enterprise application integration offers flexibility when implementing business processes; however, the cost is substantial and off-the-shelf solutions do not always fit the bill. Nonetheless, EAI is an essential element of enterprise portals.
The Institute for Healthcare Improvement (IHI) is a not-for-profit organization striving to improve the quality and value of healthcare. IHI realizes this challenging objective by targeting a range of specific needs, from improving critical care and flow through acute care to improving outcomes and reducing costs in cardiac surgery and asthma care. Its target customers are geographically dispersed and operationally varied. The organization reaches thousands of customers through training programs, collaborative projects and conferences that draw attendees from around the world. It also provides multiple channels to serve its clients, including phone, fax and Web-based registration and product sales. The multiple channel model has served its customers well but has led to application integration challenges. The most salient are:
- Different channels are supported by different applications. This is due in part to the fact that the IHI e-commerce site is hosted at a third-party site and cannot directly interface with the CRM system at IHI headquarters.
- Many details of IHI's integration plan have to be negotiated with its hosting provider. Exchanging data between databases can cause spikes in throughput on a network and the application service provider (ASP) and Internet service provider (ISP) are understandably concerned about managing those peak demand periods.
- Customer information is duplicated across multiple systems. It is not uncommon for customers to register for a course via the Web and then fax a registration to ensure the information is received.
- Duplicated information is difficult to detect because of variations in names and abbreviations (e.g., "J. Smith" and "John Smith," "1 Main Street" and "One Main Street").
IHI believed it needed a consolidated view of its customers and its transactions, regardless of how they were initiated with the company. IHI quickly realized that its in-house ETL tool, Microsoft Data Transformation Services (DTS), alone was not enough, at least as it is typically used in data warehouse environments. On the other hand, conventional EAI tools provided functionality it did not need, such as real-time messaging, and added complexity it did not want. According to lead designer Andy Hackbarth, "The EAI tools, while offering useful functionality, didn't constitute a quantum difference in capability, and so did not seem to be able to justify their expense."
IHI instead choose to develop a custom solution combining DTS as a backbone framework for ETL services, a centralized database for meta data management and a library of data manipulation services.
IHI's first pass at a custom EAI solution depended upon peer-to-peer communications between applications using DTS extract, transform and load plans. Hackbarth concluded that a peer-to-peer adapter model would be untenable for more than a few node systems and DTS was really not flexible or powerful enough a tool for what IHI was trying to do. Hackbarth realized that a centralized database with generalized representations for key entities (e.g., customers, organizations, transactions, etc.) and meta data about the data integration process would solve their problems.
Microsoft DTS provided the necessary tools to migrate data; however, many of the operations in the data exchange required complex business logic that could not be performed using only predefined DTS components. For example, customers added to the CRM system should only be added to the Web database if they have a valid e-mail address and have had valid sales transactions with IHI. When customers are added to the Web database, temporary usernames and passwords are generated and the customers are notified by e-mail of their new Web accounts with IHI. Another chronic challenge common to any organization managing customer lists is removing duplicate names. This requires business logic sophisticated enough to identify true duplicates without mistakenly removing a non-duplicated customer. Peer-to-peer comparisons of databases for duplicates are inefficient, and IHI wanted to execute the process once comparing all customer records. Utilizing a centralized database was the most effective solution to these business logic problems.
Hackbarth designed a hub-and-spoke model using SQL Server and DTS with the following components:
- Central database This instance of SQL Server is located at IHI headquarters in Boston, Massachusetts. It stores all information gathered at the various node systems (e.g., CRM tool, Web site, etc.) in a standardized format. In contrast to the EAI tools however, this database actually stores all the data, rather than reading it in, transforming it and passing it along to other systems. An auxiliary use of this database is as a marketing tool or as an operational data store to source a data warehouse. This database also provides a custom logging apparatus that is able to receive and store log messages from any DTS package. Although a centralized database can become a bottleneck in high-volume environments, this is not the case at IHI.
- Data Exchange DTS packages These are the extract, transform and load scripts between each node system and the central database. One set of DTS packages manages data flows into the central database, the other set manages outflows to node systems.
- Business logic DTS packages These packages handle the manipulation of data within the central database (e.g., duplicate search and resolution, error logging, etc.).
- Business logic stored procedures This is a library of stored procedures and user-defined functions in SQL Server that are called by the appropriate DTS packages to perform common operations across DTS packages such as adding a customer, transaction or organization to the central database; writing a log message; searching for duplicates, etc.
The central database was designed with several principles in mind. First, entities were modeled to include all attributes found in the node systems. Realizing that these systems will change, IHI added the ability to represent attributes without changing either the source or central database schema. Each entity has an associated table for specifying an arbitrary field name, type and value. This allows IHI to quickly incorporate new types of information, such as that from a partner database.
Second, source systems were not changed. Meta data about data exchanges, such as links from records in the central database back to source records in the node databases, is managed in the central database. Logging and audit controls are also handled by the central database.
Third, business logic should be reusable. Operations such as adding a customer, removing a duplicate and updating links between source and central database records are used in multiple DTS packages.
IHI incorporated a number of best practices in their application integration project. The EAI system was only as complex as needed. IHI could have purchased a commercial off-the-shelf package, but most EAI packages would have introduced more complexity without significant benefit over their custom designed solution. The system was designed for change but does not require modifications to source systems. A flexible, centralized database adapts to changes in business processes. IHI is negotiating implementation issues with its ASP and ISP. These providers are partners in core business operations and it is best if they are brought into the design process sooner rather than later to prevent mistaken assumptions that can disrupt the EAI process.
Author's Note: Thanks to Andy Hackbarth of October East Associates, Inc. (www.octobereast.com) and Paul Hamnett of the Institute for Healthcare Improvement for their contributions and insights to this article.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access