For the successful integration of strategic information within organizations, management needs to recognize the competitive advantages of such an effort. Be it greater competitive awareness, organizational synergies or more powerful analytics derived from integration, these advantages, once identified, need to be communicated across the organization.
Any integration effort then tends to be more effective from an organizational perspective and meaningful to the various players. Several visionary companies today proactively use integration and analytic efforts to spot trends, anticipate changes in the marketplace and make the necessary adjustments required to stay ahead of the competition.
It is not uncommon for organizations to have information for various functions scattered across the enterprise. This is in contrast to the actual business need of organizational information available in a central location that is easily accessible and usable by the various business functions. The impact of a lack of this centrally available information is manifold – organizations suffer from increased costs, lack of cohesive information and poor information quality, leading to ineffective decision-making.
These issues spawn further anomalies within the organization that lead to gradual systemic decline. Such anomalies could include an inability on the part of the business to be able to react quickly to changes in the marketplace, a lack of timely competitive intelligence and a lack of the ability to identify problem areas within the business.
Very often, organizations continue to function despite all of these inherent weaknesses and may have been functioning ineffectively for very long periods of time without actually realizing it. They may have been immersed in operational mode in order to meet goals and objectives with little or no time to step back and examine processes and functions holistically from an enterprise perspective. This is required periodically within organizations to streamline functions and processes and root out inefficiencies.
As the problems persist and start to impact the business negatively, a major overhaul inevitably results. The resulting overhaul usually sets about correcting these issues and we begin to see enterprise solutions start to emerge. Organizations start to right themselves in key areas, and information integration is most often right in the middle of such a change.
In order to provide an appropriate integration solution, a good place to start is with the business needs. Identify the pain points and the questions being asked by the business in terms of cohesive, centrally available and reliable information. This information is at the core of implementing an enterprise solution. Next, consider the best way to address these questions from an enterprise perspective and the best approach in terms of building integration solutions that answer questions across various functional areas.
Some features of an optimal approach are: implementation of best practices, elimination of redundant information, reusability of components and an overall emphasis on enhanced performance.
To provide greater perspective, these areas need to be examined closely in order to expose the underlying details. To start with, as mentioned earlier, business needs are an area of primary importance for which to identify information that is required. A high-level business statement might state the need to analyze the following:
- Monthly sales,
- Products sold,
- Who are the customers,
- Where they are located,
- How much each customer purchased
- Sales force, and
- Profitability.
Further analysis of this business requirement reveals the need for information around product, customer, sales force, location, time, quantity, product cost/margins and profitability.
Once the required information is identified, the next steps involve:
- Source system identification,
- Nature and format of source,
- Determination quality and integrity of source,
- Frequency of refresh, and
- Volumes to be sourced.
Examination of the source will reveal the need for scrubbing or cleansing, reformatting of structured or unstructured data and standard conversions. This analysis helps determine an appropriate plan and design of the integration effort. There are several mature integration tools available in the marketplace to choose from. These tools come prebuilt with all or most common features required for integration of information. Some of these tools provide the capability to implement more advanced concepts like real-time integration, data virtualization, parallelism and delta/change capture, among others.
Stages of Integration
For purposes of simplicity, the integration can be divided into multiple stages: information sourcing, the transformations that need to occur and the target state of the information. Each stage presents its own unique challenges and each is important in its own scheme of things. The sourcing of the information requires the analysis mentioned earlier. Volumes required play an important part. There will be instances where volumes run into millions or rows. Further analysis at this step can ease the challenge of processing times and performance. The information can be further filtered at this stage to reduce the amount of information being pulled on to the processing server. It is important from a performance perspective to apply any filtering prior to further transformations being performed on the information. If the source is relational, how is the joining performed? Joining needs to be performed as close to the source as possible.
Moving on to the transformations, it is important to achieve the transformations using as few steps as possible. This reduces the amount of processing that needs to be performed on the information. Again, some of these transformations could be achieved close to the source, making processing more efficient. As tempting and easy as it may be to use predefined constructs, it may be more efficient to achieve some transformations using native SQL language. Again, best practices need to be established around the use of code and transformations. Any written code or constructs must conform to predefined standards and be rendered reusable and easy to understand and modify. This saves development time and will eventually translate to cost benefits.
