Robert Carmichael, CTO of Solstice Software contributed this month's column.

Business Agility Requires SOA Quality

The alignment of business and IT is not just a new management philosophy but a necessary collaboration as business processes are increasingly enabled by systems and user interactions with those systems. To achieve a service-oriented architecture (SOA) environment, the technology function must first decouple its systems so individual components and services can be reformulated by the business. As the technology is recompiled, it must be thoroughly tested. While testing for the new components is similar to traditional application testing, testing the assembled processes requires new and more extensive testing approaches.

SOA Testing Best Practices

SOA provides a service ecosystem in which composite, business-centric services can be assembled on the fly in response to required changes. In a SOA environment, a change in one part of a business system is no longer an isolated alteration. Unfortunately, application testing does not totally satisfy SOA testing requirements. With an emphasis on application logic and the client interface, application testing is inwardly focused. SOA testing also needs to be oriented outwardly toward the business process flows connecting components, services and data in order to complete end-to-end testing. What does SOA demand of quality assurance best practices for a process-driven environment?

  • Visibility
  • Coverage
  • Collaboration
  • Automation


Traditional application testing assures that the user interface performs as expected. SOA testing needs to go behind the screens to expose the entire assembly and the data driven through the process. For example, a process may be initiated from an organization's Web portal to go to legacy systems, crossing over to a third-party application for necessary data, be screened by another application and passed through a client profile management system to record the transaction, and then complete the process through the legacy systems again before returning the result to the front end.

In some systems, instrumentation is used to measure quality of service (QoS) metrics. However, these capabilities still do not provide complete end-to-end visibility. Most organizations end up writing test stubs and harnesses and creating queue dumps to generate diagnostic information. These methods are not only constrained by their manual nature but also by their piecemeal nature. Development teams are usually asked to write the code for each component or application, leaving it to a project team to string these pieces together and assemble testing results.


Test coverage is a well-accepted aspect of testing discipline. How is coverage different in a SOA? Today, many SOA processes are being developed as headless applications that do not require direct user interaction. In this case, traditional application testing is inadequate. Best practices should look for ways to maximize process visibility and use that visibility to maximize process coverage. Further, because SOA coverage must consider how each element operates within a larger process, not just how an element operates in isolation, SOA testing best practices need to assign risk weights to each element as part of the larger overall process.

For example, Gartner has observed that traditional coverage ratios get diluted when considering the process as a whole. For instance, if traditional coverage is set at 80 percent for each part of a three-part process, the overall coverage equates to only approximately 50 percent (80 percent × 80 percent × 80 percent). Therefore, techniques need to be added not only to improve the coverage of all the hidden elements, but also to raise the assurance level by covering the end-to-end process itself. In SOA, the connectivity makes each element critical for coverage. The lack of process visibility contributes directly to a lack of process coverage.


Testing today for complex, cross-functional integration projects can be terribly inefficient. Organizations recount stories of meetings involving 15 to 20 employees and contractors, sitting with piles of dump reports and boxes of pizza to sustain them through the tedious exercise of culling through data in the hopes of uncovering the root cause of a bug. Best practices must transfer quality assurance (QA) tasks to QA testers, operations and developer staff that are empowered with the visibility to drill down into data across the entire integration project process. Process matter experts (PMEs) and business analysts are also part of the team that now owns quality and have been added to organizations committed to business process engineering and SOA. As specialists, they bring a top-level perspective of the ramifications of end-to-end processes. Testing is thus enabled through a form of collaboration across roles that are chartered with assuring a consistent deployment of the integrated business process.


For complex business processes, testing accuracy and speed are dependent on the ability to rapidly assemble SOA components. As such, testing best practices for SOA will benefit by adopting an assembly-phased process. Component interfaces should be tested as they are added to the overall integration architecture. Organizations using agile development and other test-driven development methodologies should be able to transition easily to this testing approach for process validation. If the process requires systems to be available for complete end-to-end testing that are unavailable or not built, this problem can be reduced significantly by simulating missing components in the testing process.

Compatibility testing underscores another difference when it comes to SOA testing. The concept of process testing is expansive, including the comparison of a process to its former version (or "baseline"), the impact of both intended and unintended changes, the impact of an addition, what-if assessments and the overall business impact of the process within an SOA framework. As demonstrated throughout this article, the sheer scope of SOA testing demands automation. Test management has become more important as scope and number of tests increases. Here again, SOA integration testing best practices should incorporate tools and methodologies that have been created to help organizations execute complete system testing (application and integration).

Assuring quality business processes through SOA, requires the incorporation of quality assurance best practices that manage the key elements of SOA integration testing: visibility, coverage, collaboration and automation. What sets these practices apart from traditional testing practices is the focus on the end-to-end business process and the dynamic nature of testing within a continuously changing environment such as SOA.

Robert Carmichael joined Solstice Software in February 2004. He is Solstice Software's CTO and is responsible for driving the product roadmap, overall product quality, and leading the software development of all Solstice products. Prior to joining Solstice, Carmichael served as VP of Engineering at Catapult Technology. He was a major contributor to the growth of a consulting practice specializing in Enterprise Application Integration (EAI) implementations, leading to the acquisition of Catapult by Idea Integration in 2000. Carmichael has worked for early stage companies in both product development and product management roles and has had exposure to a broad range of technical architectures and solutions, specializing in the EAI space. He has a B.A. in Economics and Applied Mathematics from Rice University.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access