With continuous change increasingly driving businesses to become more agile, time is money and having access to critical business information at the right time is key to maintaining competitive advantage. This situation is becoming even more complicated as businesses are becoming increasingly impacted by powerful forces such as compliance, globalization, increasing competition, improved customer service needs, shrinking budgets and M&As. If we add to this already daunting list of challenges, the ongoing explosion and fragmentation of data all across the enterprise and beyond further complicates the situation. To support such dynamic environments, enterprises are increasingly pressurizing IT to deliver holistic and accurate business-critical information at the speed of the business.
In recent years, service-oriented architecture (SOA) has emerged as a leading technology for enabling a new generation of more flexible and cost-effective IT solutions. SOA promises to deliver business agility by breaking down barriers between silos of applications with flexible and reusable business services. However, if the data stuck inside silos of applications is bad, imagine the calamity when, through application and business process integration technology, the silos disappear and data from many different applications is commingled. SOA seems to have overlooked the data that feeds these processes and applications. Technologies such as enterprise application integration (EAI) and enterprise information integration (EII) have fallen short of providing flexibility in data latency and volumes.
This article introduces you to a data services platform as the most efficient approach for enabling business agility across the enterprise, through the delivery of right-time information, be it information delivered in batch, near real time or real time. With a data services platform you can enable scalable access, integration and right-time delivery of business-critical information to enterprise-wide composite applications. A data services platform maximizes business value using right-time information for driving competitive advantage, lowered risk and cost-effective project implementations.
The availability of information at the right time directly impacts business agility, making it an extremely strategic enterprise asset. Businesses must continuously look into maximizing the strategic value of information by overcoming the roadblocks that stand in the way of leveraging timely information for business benefit.
Business concerns such as the ability to continuously differentiate through better decision-making, keeping costs under control by doing more with less, successfully executing M&As by rapidly on-boarding new systems, easily complying with changing laws through information-centric compliance reporting and delivering improved customer service by providing timely, accurate and consistent information, all point toward leveraging information as a strategic asset.
As business requirements are continuously changing to drive increased agility, IT departments are finding that they need to become more responsive than ever before. However, in trying to support business needs more efficiently, concerns abound as to how to efficiently deal with complex IT infrastructures involving diverse and distributed data sources. CIOs, IT managers and architects are often faced with addressing challenges such as:
- Heterogeneity of data sources distributed across the enterprise and beyond,
- Inconsistency of data and constantly changing data structures,
- Poor data quality that is often difficult to measure or monitor,
- Lack of agreement or visibility (single-view) into business-critical information assets, and
- Little to no re-use of data integration logic and skills.
Data silos pose a real threat to businesses looking to efficiently leverage information when needed, as enterprise data is typically fragmented, inconsistent, inaccurate, lacks a single version of truth and is extremely complex to deal with.
Some of the typical integration issues and consequences impacting the availability of information at the speed of business that may exist in the enterprise include those listed in Figure 1.
As we can see from Figure 1, having to deal with multiple and diverse data sources leads to a very complex IT infrastructure that can in turn lead to delays in getting products and functionality out in the market and hence affects business agility. Other issues, such as brittle architectures, lack of a single view of data, limited change management and duplication of data integration efforts, can be powerful inhibitors for right-time information availability, hence directly impacting business agility.
The complexity of enterprise data - its volume, its latency, its many formats and structures - requires that data be treated as a strategic enterprise asset that addresses the various data integration challenges in the enterprise and be delivered as a service.
In order to leverage right-time information to deliver business agility, the underlying technology must ensure the following:
- Seamless handling of any data structure, data format and data access mechanism without much heavy-lifting;
- Providing a single and consistent view of all enterprise data across all data sources spread across the enterprise and beyond;
- Insulating data consumers against changes to underlying data sources and protecting the IT infrastructure from brittleness;
- Lowering IT complexity by limiting points of maintenance of interactions with the underlying data sources;
- Reducing inconsistencies and inaccuracies across all data sources available in the enterprise;
- Supporting complex data integration tasks such as data access, data profiling, data cleansing, data transformations and data delivery;
- Enabling data lineage and impact analysis for effective change management and compliance reporting and reduction of risk;
- Promoting the re-use of data integration logic and skills across all various lines of business and projects; and
- Deliver business-critical data at the speed of business (batch, near real time and real-time).
Solution Approaches and their Limitations
At this point, as shown in Figure 2, lets take a moment to review some existing data integration approaches such as hand coding and EAI. In this section we will examine why they are either inefficient or complex solutions that have limited or no support for a number of the key requirements for maximizing the strategic value of right-time information.
The most common approach to addressing data integration issues in the enterprise is hand coding. Hand coding involves the manual integration and handling of diverse data structures, formats and access, with little or no protection against changes to the underlying data sources. There is also no way to handle data quality issues across various data sources, and because this is not a standards-based approach, there is limited extensibility reuse across projects. Further, there is no way to know the origin of data or how data is being used and sophisticated data integration tasks such as transformations require significant manual coding efforts, which leads to elevated levels of ongoing maintenance.
While EAI is potentially a good approach for tackling application integration issues, it is not the best approach for addressing challenges that lie in the path of delivering right-time information. There is no support for bulk and trickle-feed data movement, and there is no easy way to automatically detect changes in the various data sources. There is also a lot of complexity involved in handling data quality issues, and with little or no reuse of data integration logic across use cases, it can be a rather expensive approach. Further, there is limited support for metadata and impact analysis and sophisticated data integration tasks such as complex data transformations.
Both hand coding and EAI distract from the core purpose of creating differentiated business logic, which along with increased infrastructure complexity, limited reuse of data integration logic and ongoing maintenance costs, begs for a more efficient solution.
The Ideal Solution - Data Services Platform
So, what is the ideal solution to the highly complex data integration problem that can minimize the complexity and cost of enabling the availability of accurate and consistent information at the right-time?
The answer is data services. As shown in the figure, data services is a highly flexible, simple and cost-effective solution that provides a model and standards-based reusable abstraction layer that lowers the complexity of integrating data silos and delivers a single, consistent view of all enterprise data at the right time.
More specifically, a data service is a modular, reusable, well-defined, business-relevant service that leverages established technology standards to enable the access, integration and right-time delivery of enterprise data throughout the enterprise and across corporate firewalls.
Data services deliver the following compelling benefits:
- Reduced complexity by insulating data consumers from dealing with underlying data sources,
- Increased time-to-market of new functionality by isolating applications from data sources,
- Lowered brittleness and maintenance by providing an abstraction layer to all data sources,
- Enabled extensibility by simplifying access to data sources by a standards-based interface,
- Renewed focus on developing business logic by seamlessly handling data integration issues, and
- Lowered costs by promoting reuse of data integration logic and skills across various projects.
When the inherent capabilities and benefits of data services are delivered by a sophisticated and scalable platform, we are looking at a powerful solution that can maximize the business value of right-time information and ensure business agility.
To efficiently enable the availability of right-time information, a data services platform must provide the following functionality:
- Provide a codeless data services development environment that can promote reuse;
- Scale intelligently and automatically to support increased concurrent data services requests;
- Deliver sophisticated data integration services to address the entire data integration lifecycle;
- Support the combination of individual data services into meaningful data integration logic;
- Enable a way to proactively address data quality issues across all data sources;
- Provide a broad range of data access seamlessly through a range of available connectors;
- Deliver right-time (bulk, trickle-feed and real-time) information with a single optimized engine;
- Ensure the quality and consistency of data across multiple and disparate data sources;
- Use a metadata-driven architecture to standardize and reuse data definitions
- Leverage a common metadata repository for enabling data lineage and impact analysis;
- Deliver business-critical information to a variety of data consumers in the desired format; and
- Provide a complementary technology to IT architectures and integration competency centers (ICC).
The Data Services Platform and ICC
An ICC is an infrastructure of people, technology, policies, best practices and processes focused on rapid, repeatable and cost-effective deployment of data integration projects critical to meeting organizational objectives. Organizations have found a direct connection between the caliber of their ICC and the companys ability to respond quickly to dynamically changing business models, intense competition and demanding customers. In short, the ICC is the infrastructure responsible to deliver trustworthy data flexibly and in real time.
A data services platform encourages the creation and management of ICCs by enabling the following:
- Standardizing development processes on common technology standards, enabling greater reuse of work from project to project;
- Supporting a shared services environment in which a centralized team maintains shared work and environment, but most development work occurs in the lines of business;
- Supporting a central services environment in which a centralized team is responsible for all development work on integration initiatives; and
- Supporting governance processes of data services and data architecture such that definitions, semantics and software license agreements are maintained by the appropriate parties.
A sophisticated and enterprise-grade data services platform supports the rapid development of scalable data services to deliver holistic and accurate information at the speed of business. With businesses continuously looking to become more agile, a data services platform is a perfect complement to an overarching enterprise integration infrastructure to deliver right-time information to all data consumers.
Register or login for access to this item and much more
All Information Management content is archived after seven days.
Community members receive:
- All recent and archived articles
- Conference offers and updates
- A full menu of enewsletter options
- Web seminars, white papers, ebooks
Already have an account? Log In
Don't have an account? Register for Free Unlimited Access