Disparate data, duplicate data and inaccessible data are all serious problems that can cripple an enterprise. Everyone wants to address them, but budget-conscious corporations cant afford to blow millions of dollars on a project that will disrupt business for long periods and may break within months or weeks of finally being put into place. The sky-high cost of integration projects and timelines measured in months or even years make them the dread of many businesses. If youre unfamiliar with the complexities of integration projects, transferring data back and forth between a few applications, file formats or data stores may sound deceptively straightforward.
Why are Integration Projects so Expensive?
Many factors make integration expensive some are fairly obvious, some possibly surprising. A Yankee Group report from a few years back, The Hidden Costs of Data Integration, uncovered some less obvious contributors.1 The reports main conclusion was that while businesses budget for building or purchasing integration tools and setting up initial integration implementations, they rarely factor in long-term maintenance and repair costs. The report notes, Over a three-year period, the total cost of ownership of an integration application is more than eight times the initial software license investment.
Budget Busting Factors
Staffing costs. According to the Yankee Group, labor is the most expensive aspect of an integration project, costing as much as the software and hardware costs combined over three years time. This cost as well as the costs associated with disruption of the normal business routine increase as implementation time increases. Anyone who has been involved with integration projects knows that implementation times frequently turn out to be higher than expected. The more those schedules get stretched, the higher the price tag.
One reason for high staffing costs is that the business environment is highly unstable due to mergers and acquisitions, changes in business strategy, new market initiatives and other factors. This instability means that even the most robust integration implementations have to adapt constantly. As much as labor costs incurred during implementation can put a dent in the budget, the costs of making changes can be even more extreme. The periodic downtime of a system that the corporation has come to rely on can cause myriad unpredictable problems that have their own high price tag.
Data quality. Data quality issues are another major cause of system downtime and implementation delays. Even data that everyone involved believes is clean and rigidly regulated frequently has flaws that can cause integration pipelines to crash, driving up downtime and repair costs. If inaccurate or improperly classified data makes it into mission-critical data stores, the result can be faulty analytics, damaged trust and lost business, each of which is costly.
Endpoint variety. One of the biggest contributing factors to integration cost is the wide variety of data to be integrated. Endpoints may include legacy systems on mainframe platforms that have been operating for 20 years or more, while others are software as a service (SaaS) systems that were recently implemented on platforms outside the firewall. In between are a vast array of applications of all types and ages on a variety of platforms with myriad interface types. Because no two programmers solve data storage problems in quite the same way, no two applications ever store their data in the same structure or with the same semantic logic. Even two applications of the same type, like two customer relationship management (CRM) systems or two accounting systems, can show extreme variations in data storage strategies, making the data incompatible without extensive transformation.
Project complexity. The demands of modern integration initiatives increase the level of complexity of the problem, even while they seek to simplify and improve business processes. Frequently, simplicity for the business user translates to more complexity in the underlying architecture. Data warehouse service-oriented architecture (SOA), business intelligence (BI) and master data management (MDM) projects all propose to make business processes more efficient by connecting multiple disparate systems, but they require complex initial implementations to reach their goals. Meeting the technical demands of real-time updates, bidirectional synchronization, and exponentially increasing data volumes doesnt come cheap.
With multiple factors conspiring to make integration projects into budget busters, its amazing that anyone accomplishes them at all without ending up in bankruptcy court. Yet, thousands of companies have accomplished very successful integration initiatives, stayed within a reasonable budget and ended up with a much higher ROI than their initial investment. Implementing a solid budget-conscious integration project isnt impossible. You just have to use the right strategies to counter the factors that make integration expensive.
Create an Integration Strategy That Will Not Break the Bank
Adjust your expectations. The first thing that has to happen is that you need to adjust your expectations, particularly of integration tools. Ridiculous price tags for integration projects should not be standard. Bloor Research recently did a survey of approximately 200 companies comparing various integration vendors for price across multiple projects.2 There was a huge difference in initial license costs for integration software, from hundreds of thousands to free for open source tools or those included with other software. Strategies that companies might think would save money are:
- Invest in more expensive software up front, based on the idea that it would deliver ROI by being more full featured and easier to implement, or
- Go with open source software or other software that is virtually free and save on up-front licensing costs.
The Bloor study compared various integration tools, including the other factors needed to make those tools work, such as labor, hardware and dependent software requirements. Some of the results were surprising. In particular, open source software, due to high technical resource demands and low reuse potential, was not necessarily the most inexpensive across multiple projects. The high-dollar software was often just as costly long term as it was up front, as much as doubling the cost-per-project relative to other options.
Start small and scale up. This strategy can be summed up as think strategically, act incrementally. Big-bang projects that seek to solve all of a large enterprises integration issues at once can take months or even years to implement and have a notoriously high failure rate. The enterprise must struggle with makeshift patches and constantly shifting applications. By the time massive projects are completed, the shifting landscape has frequently made them obsolete. One key to making integration cost-effective is to build incrementally. Choose a single pressing business issue, and solve just that. Be sure to choose the technology, the tools, the architecture and what challenge to tackle first based on the business need. Dont make the mistake of choosing what and how to build based on the technology.
It does make good sense to plan ahead, though. Think of the long-term, overarching architecture, as that will give a consistency of vision that can deliver huge benefits later. When it comes to actual implementation, start small. Get an immediate ROI, get visibility into how integration can benefit the company, and learn what works well for your particular enterprise from that first baby step. Use that knowledge to tackle the next business issue. This allows integration to grow organically along with the business needs.
Extend existing solutions. Theres an old saying, If it aint broke, dont fix it. Integration pain is frequently solved, at least partially, in an ad hoc fashion by the people who feel the pain on a daily basis when trying to get their jobs done. SQL stored procedures, Perl extraction scripts, or little custom-coded executables can all be used as patches for the big holes between applications. These stop-gap measures are rarely sufficient by themselves, but sometimes accomplish their limited goals reasonably well. Dont waste time and money ripping and replacing integration aspects that are functional for their limited purpose just so that they fit the new architecture. Many integration tools can call out to those procedures and executables and incorporate them as small components in more complete, robust business processes. Whenever you can, save time and money by using whats working, incorporating it into new projects that fix real problems and add new value. Later, as time permits and performance needs demand, those stop-gap measures can be updated to fit in the new architecture.
Reuse as much as possible. Having an overall roadmap of where you want to go gives you the chance to plan reuse of components. Even if a small point-to-point project is all that is on the table at the moment, choose tools and architectures with reuse in mind. Point-to-point integration today may lead to a data warehouse, MDM project or SOA later. Dont set yourself up to reinvent the wheel and buy a whole new toolset for each integration project. Build with reuse in mind and, when those next projects happen, reuse existing tools and code wherever possible to save time and conserve resources.
Select technology carefully. The choice of tools and architecture is critical, but keep them as application- and platform-agnostic as possible. Applications change and companies shift. The more agile the integration platform, the more flexible the integration solution, and the more it will be capable of dealing with change. Choose an architecture and toolset that can handle diverse endpoints and a variety of execution platforms to deal with whatever the future of a changing business throws at you. A consistent integration platform can simplify some of the complexities of implementation across time. Also, a single common toolset that can be used for multiple projects gives the team the advantage of tool proficiency from the first incremental projects to shorten later project time-to-value. The checklist at the end of this article can help ensure that youre selecting technology with cost factors in mind.
Know your data. Wags refer to the standard method of integration implementation as code, load, and explode: creating an integration pipeline, sending data through and waiting to see if anomalous data will crash the system. This causes a lot of implementation deadline delays, as well as a lot of breakdowns after initial implementation. Thorough automated data profiling, preferably of all fields in all incoming records, gives advance insight into any problems that could cause pipeline failure. Profiling the data up front can also help you deliver realistic integration project time estimates and prevent costly surprises.
There is a cost, both monetary and temporal, for putting into place thorough up-front data profiling. Many organizations think skipping that step can help save on integration projects. But the cost of errors found at the beginning of the integration process is far less than the cost of data errors that are allowed to become a wrench in the gears or misinformation in the final data stores. Data knowledge is not a good place to skimp. Catch data problems up front, and build automated remediation or other data error handling into the system as much as possible. In the long run, profiling will save huge amounts of resources, as well as providing more reliable and trustworthy data, which translates to a higher ROI for the integration project as a whole.
Build with change in mind. A flexible solution is key to keeping costs down over time. Expect and plan for change. Building a rigid integration solution that will break and have to be rebuilt when changes happen just doesnt take business realities into account. Business and IT environments are never static. Applications are upgraded or replaced, businesses merge and grow, data volumes increase and new endpoints appear. Implementations that dont plan accordingly are guaranteed to make a significant dent in the corporate pocketbook. Some of the most essential elements to keep in mind for a flexible solution are robust built-in error checking to catch problems quickly, fast visual toolset design interfaces for repair and adaptation to change, and isolation of volatile endpoints from business processes so one change doesnt break the whole system.
SOA architectures have much to recommend them, especially for larger enterprises. They provide a high potential for reuse of integration business process flows and a level of abstraction between business processes and applications that enhances flexibility. However, SOA is not a panacea. If all that is needed is a single point-to-point integration, building an entire SOA infrastructure to make it happen drives up the price tag unnecessarily. Be sure to evaluate the advantages of a data service model carefully for your particular enterprise.
Heres a basic checklist for cost-effective integration that wont break the bank either during implementation or over its lifespan:
- Plan for the future, but start small and scale up incrementally.
- Build for reusability and reuse existing components whenever you can.
- Extend new architectures by incorporating existing solutions.
- Incorporate data profiling and remediation up front.
- Build a solution that adapts to change with error checking, rapid repair toolsets and business process isolation.
- The Yankee Group. " Uncovering the Hidden Costs in Data Integration." The Yankee Group, 2004.
- Phillip Howard. " Comparative Costs and Uses of Data Integration Platforms." Bloor Research, August 2008.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access