This month's column is contributed by Steve Craggs, European chairman of the Integration Consortium.

Service-oriented architecture (SOA) is a subject occupying the minds of many companies across the world. Businesses are desperate to achieve the many benefits promised by this model for IT resource architecture and interaction, such as improved responsiveness to business requirements, reduced costs and lower risk. Sadly, although the basic concepts of SOA are relatively easy to grasp, achieving SOA success is something that some companies are finding to be elusive. It is as if the SOA promised land, where IT components are turned into building blocks of business functionality that can be assembled and reassembled quickly and easily to meet changing business needs, is tantalisingly visible but always in the distance. Companies are faced with the same challenge as John Bunyan's allegorical hero as he tried to reach the Celestial City, having to choose between the seemingly easy road and the harder, narrow path. Unfortunately, as in the story, the easy way is often the wrong choice.

SOA Recap

Before looking at the different approaches to SOA, it is worth recapping the key SOA characteristics that are fundamental contributors to the desired benefits. First, components in an SOA should be independent and loosely coupled. The caller of a particular service should not need to be aware of anything about the technology environment in which the service is running, in effect turning components into easily reusable assets. This leads to two more key characteristics, the ability to achieve much higher levels of reuse and reduced redundancy through that reuse. Not only do both of these benefits lead to a reduction in costs, but they also lead to a far more responsive IT environment with greatly reduced time to market for new projects. Another key characteristic is the ability to make the most of existing assets by turning them into reusable services, in addition to addressing newly created ones. And of course, the fact that SOA services are defined to relate directly to business functionality makes it possible to align IT strategy much more closely to business goals.

It is these characteristics that, in the end, distinguish the right SOA path from the wrong one.

The Wrong Road to SOA

The issue facing all companies, once they decide that SOA is the right goal for them, is how to proceed in developing and deploying a service-oriented architecture. To many, the answer is obvious. It is generally acknowledged that Web services and associated standards form an excellent technology for enabling an SOA. Web services offer a standard, clean way to drive different IT components and, therefore, are a valuable tool in building SOAs. There is also the added benefit that Web services can run over readily available communications options such as HTTP.

However, these two features of Web services form a seductive mix for some companies, who conclude that this is all that is required to build and deploy an SOA. All that needs to be done is to turn all IT programs and components into Web services, then link them across an HTTP connection. As more and more components become Web services enabled, the SOA will expand to eventually cover the entire business, yielding all the benefits already discussed.

But what is the reality? Companies that have tried this approach to SOA have experienced mixed fortunes. Typically, results are impressive initially, with the first few services being developed and deployed quickly and efficiently. However, as the SOA grows, cracks begin to appear in the attractive façade. Reuse levels start to fall back, levels of service drop and eventually companies are left with yet another legacy structure that is just as inflexible and resistant to change as before the SOA was introduced.

There are a number of reasons for this deterioration in SOA returns, most of which relate to the creation, deployment and operations of the services. The first problem is that once the programmers have worked out how easy it is to develop and publish a Web service, they do so without always thinking through the implications. For example, one programmer writes a service and then tells his circle of colleagues about it. Others might start to reuse it in their own applications, but there is no control over this reuse and gradually anarchy sets in. Programmers working on other projects may not know about the new service, or might find that it doesn't do precisely what they want, and therefore they might end up writing their own version. Or, they might find out about it and start reusing the service without anyone else knowing. The result is chaos, where services are spread around the enterprise with duplicate occurrences, and where no one is sure which services are used by what application. A future change to a service may have unexpected impact on other parts of the business as a result, and duplicate copies can start to diverge as modifications are made.

Another problem is that as SOA starts to play a key role in critical operations, the inadequacies of the transport mechanism can cause problems. HTTP does not offer guaranteed, once-only delivery of information from one component to another. This may not matter in some business scenarios, but in others it may be catastrophic. Transferring funds is an example of an operation where an accidental duplicate request could have serious consequences. Web services fanatics will say that new standards such as WSReliableMessaging are being developed to address these issues, but these standards are far from mature and expecting them to address this type of issue is frankly a pipe dream. Someone still needs to actually implement the standard, after all.

Then there is the question of service levels. Web services expect information to be provided in XML format and have related WSDL that must be interpreted. However, as more and more business function steps are turned into services, there can be problems with constant mappings from non-XML to XML formats and network trips from one environment to another. For example, because many mainframe environments do not deal in XML, any services invocation of a mainframe program requires XML to non-XML mapping and then back again, as well as a hop to the mainframe and back. As the number of services grows, this can become a serious issue.

Some people argue that they have managed fine with this simplistic approach to SOA, but generally these examples involve only a limited number of service deployments. When talking about 10 services, managing the usage of them is not that hard - but once the number rises to 100 or more it is a different story.

So how can a company achieve sustainable SOA advantage? The answer is to bite the bullet and opt for the harder, narrow path. It is harder work but is far more likely to lead to SOA nirvana.

Taking the Right Road

In truth, there is no magic here. The right path involves hard work and putting in place a set of sound processes and procedures, both at the management and technical level.

The Technical SideStarting with the technical side, which is in some senses the easiest, there is nothing wrong with deciding that Web services is going to be a fundamental building block of the SOA implementation. However, what is needed is much more than this. In fact, the best bet is to put in place an entire SOA ecosystem. Figure 1 outlines the main constituents of this ecosystem.

Figure 1: SOA Ecosystem

Obviously, the services lie at the center of the ecosystem. However, there must be support for custom-built services (both brand new and existing components exposed as services), external ones and ones provided by packaged software. Around the services are areas of software designed to support the creation, deployment and usage of these services.

The registry layer provides a directory of services and related information. It needs to be accessible to all nodes in the SOA and, as such, will probably need to be federated for performance and perhaps mirrored to avoid any single point of failure. But the most important aspect of the registry is that it needs to be really easy to use. Programmers have a natural tendency to want to write things from scratch - it is more fun and you know what you will get. So, if reuse is to be encouraged, it must be extremely easy and quick to identify what services already exist, what they do and how to invoke them. In addition to this, the registry layer can also add a lot more value by keeping a record of all other related information such as whether this service requires encryption/decryption of input and output, if it needs to be recoverable and local information such as the name of the service creator. Finally, because the registry holds information on all services, it provides the ideal location for intelligence about usage of the services, such as which applications are using what service and what version of service is the current one.

The communications layer represents the mechanism for connecting different nodes together, and to avoid some of the quality of service drawbacks of pure HTTP it may be useful to use some sort of commercially available message-oriented middleware such as a Java Message Service (JMS) implementation. The gateways and adapters layer fits closely with this, providing software to make it easy to interface to a particular technology environment. The most common requirements are to get at various packages, or homegrown legacy environments. For example, some tools offer "wrappers" for mainframe CICS or IMS applications that make them look like Web services.

Also related to communication is the mediation layer, which provides added value to inter-component communications as the information flows between nodes. One common mediation function is transformation, dealing with the need to map data stream formats to ensure that every component can receive and transmit information in its native format, with the format being switched as required. Another is orchestration, where a particular service linkage flow may be adjusted based on business rules and policies. Others include data enrichment and logging. Frequently, companies select an enterprise service bus (ESB) to deliver the support that makes up the communications and mediation layers. This can even address some of the functionality in the gateways and adapters layer.

Development tools need to be considered in two distinct categories, although some tools may address both sets of requirements. There is a need for tools to help with the creation of the services themselves, and also for tools to create and define the SOA infrastructure. An important factor with the service creation tool is that it needs to be as intuitive as possible. Skills problems often loom in SOA projects, particularly when it is necessary for components in unlike environments to be integrated. It is almost impossible to get a mainframe programmer to understand UNIX or vice versa. Instead, the tools need to ensure that both the mainframe and the UNIX specialist can become service developers, so that other environments can use services without any need for cross-skills training.

Finally, operational tools will be required to deal with such requirements as tracking the operational aspects of the SOA, looking at services performance and usage, identifying out-of-line situations and a range of administrative and governance-related activities.

Providing this ecosystem is unfortunately not trivial. There is no one vendor today offering the ideal functionality across the entire ecosystem. Most companies are using a best-of-breed approach layer by layer. Because of the standards-based aspects of SOA, there is generally not too much difficulty fitting the selected tools together to form the overall ecosystem.

However, supplying the ecosystem is a necessary but not sufficient condition for sustainable SOA success. There are management aspects that must also be considered.

The Management SideOne of the most important factors to appreciate about SOA is that it is not just a technical initiative - it is also a business transformation issue. SOA can deliver greatly enhanced business agility, but only when ways of doing business change.

For example, a frequent reason for the emergence of duplicate Web services is the fact that different departments do not talk. So, for example, one department decides it needs a "get customer details" service, and builds it. Another department, needing the same service, decides that it needs different information to make up the customer details, and writes its own version. Or one department reuses another's service, only to be hit in the future when the "service owner" changes it without the using department's knowledge.

The answer is to establish processes and procedures whereby business services are specified, defined and agreed across the enterprise and are subsequently managed and controlled. This can be quite a challenge for many companies - escalation processes will be required to resolve conflicts, and everyone involved must be made to understand that cooperation is in the best interests of company performance. This requires heavy management involvement, probably being driven from senior levels. One critical aspect of agreement must be the level of service granularity. For performance reasons, it may not be a good idea to turn every program into a service. In the previous example of components split between UNIX and mainframe environments, suppose there are three mainframe tasks and one UNIX one that need to be executed to deliver the required business service to the UNIX programmer. Turning everything into Web services will likely cause performance issues because of XML to non-XML conversions and a lot of network hops. Instead, it may be better to build a single mainframe Web service, with orchestration underneath to run the three mainframe tasks. This will prove far more efficient.

Another issue to be handled is to change the mind-set of the developer toward reuse. Some companies handle this by actually incenting developers to reuse services with cash bonuses based on the percentage of reuse they achieve. Once a good selection of services is available, a new type of programmer role appears where a business analyst might assemble a new business process by simply combining different service building blocks together into a composite application.

There clearly are at least two paths to choose when implementing a service-oriented architecture, but unless usage is likely to be extremely limited and light, the approach most likely to succeed is the harder, more thorough one. Companies should consider the SOA ecosystem requirements, make the appropriate tools selections and put in place the necessary management measures to ensure that the SOA implementation will be a controlled and future-thinking project. Management buy-in is essential, with a number of potentially tricky obstacles to be overcome. However, if sufficient planning, care and attention are applied in conjunction with a suitable ecosystem, companies will find themselves on the right road to SOA paradise, where all the SOA benefits are real and sustainable.

Steve Craggs is founder and president of Saint Consulting Limited and European chairman of the Integration Consortium.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access