I consulted with a major financial services company that wanted my opinion on a dashboard they were building for their CFO. This dashboard would have a real-time tachometer showing each instance when a journal entry transaction was entered into one of their 11 general ledgers around the globe. Curious, I asked about the reasoning behind the functionality, and their answer was that the CFO wanted more up-to-date information. In reality, the CFO needed midmonth data to help understand why one system was always running behind in closing out their monthly books. The CFO was not asking for real-time data, he was asking for right-time data.

With advances in technology and the need to shorten the time to decision, organizations are moving beyond the traditional nightly or weekly load into more real-time and near real-time applications. These developments in advanced data loading and real-time integration are exciting and applicable to many business situations, but seven best practices must be applied to fully leverage the technology.

1: Understand the Business Need and Timing

Any application - whether BI, performance management, planning, reporting or analytics - should start with the business problem at hand. Before any code is specified or written, the business requirements must be modeled, interrogated and approved by the business sponsor.

A solid understanding of the data, presentation, business rules and consumption is necessary. Whether this is done through use cases, prioritized lists, current report analysis or some other method, it needs to be done.

Getting signoff on the requirements is equally important. We all realize that requirements will need to evolve in the business, but getting a finite, point-in-time signoff will support selection of a technology platform that addresses the known requirements.

Every piece of business data has a shelf life. After a certain point in time, business data is not useful for decision-makers. For example, in today's economic environment, there is a need for much more up-to-date customer credit information to decide whether to extend lines of credit. Quarterly data that was previously acceptable may be required on a monthly or weekly latency instead. Another example is inventory levels. As organizations manage their inventory and cash flow more closely, a more up-to-date inventory level may be required for decision support systems.

As with business needs, the people who best understand the timing requirements are the business leaders and users. It is very tempting for IT to gather the needs and then apply the most advanced tool in its arsenal, such as real-time integration. Instead, IT must be patient and get direction from the business. And just like the data requirements, IT needs to get signoff on these requirements.

By skipping this step, the financial services company mentioned earlier would have incurred additional costs, increased the project's time frame and risked jeopardizing the relationship between IT and the CFO.

2: Model for History

In a traditional data warehouse environment, the main decisions about history were typically a) which type of slowly changing dimension was to be used and b) what the budget was for storing this history. With real-time and near real-time integration technologies, the BI application often connects directly to a data source and does not physically move the data into a warehouse or mart. Because transaction systems are dynamic by nature, the capture of the various historical points of the data is at risk of being lost. This may be acceptable to the business because it may not care about the intermediate results and only want end-state data. For example, if the business need for end-user reporting on a 401(k) is to report on the various positions in equities at the end of the day, the multitude of trades and reconciliations that happen throughout the day may not be necessary in the decision support system.

3: Be Cautious of Data You Don't Own

Decision support systems typically will need internal data that resides within an organization's systems and external data from outside parties, such as business partners or data syndicators. Although the business may need this data in a low latency fashion, there may be business or technical boundaries that will not allow a movement from high latency to low latency. IT must understand the contractual as well as technical constraints with regard to external data.

I worked with a product manufacturer that was purchasing market share data to gauge their relative position in the market. This manufacturer wanted to test a few new products and correlate this with the market share data. They wanted to gauge market penetration on a weekly basis to decide whether to formally launch a product as part of their mainstream product line, but they were constrained by their market share data provider, which only provided monthly aggregate data that typically lagged by one month. All of the manufacturer's technology and middleware could not change the reality of the constraints of their data provider.

4: Be Prepared to Evolve

Just as business requirements will evolve, so will an organization's technology choices. It is critical that IT incorporate the ability to easily move from high to low latency and switch between traditional "lift and shift" and more real-time integration technologies when selecting and building an architecture. IT needs to avoid the temptation to hardwire into the existing systems. The use of middleware and SOAs will mitigate against the hard-wiring issue.

5: Prepare for a Technology Shift

Many of the organizations I work with have been quite successful in implementing various data marts and data warehouses. They have acquired the technologies that are right for their particular application and business functions. They have also invested in their staff's ability to work with their various tools to support the loading of the warehouse and marts.

Although organizations should look at leveraging existing investments in their tools and architecture, the move toward the hybrid low/high latency environment will most likely require an acquisition and/or repurposing of existing technologies. Organizations will need to guard against trying to fit a square peg in a round hole because they are comfortable with a certain technology. Moving to right-time mode requires IT to sometimes get out of their comfort zone created by traditional high latency warehousing projects.

Organizations also cannot underestimate the human impact. There will be people in the organization who have made their reputation by being the master of the data warehouse and may resist the move to a hybrid approach. This is particularly true in larger organizations where the middleware and integration group may have competed for internal projects with the warehousing group. I worked with a global life-science company with one group that provided middleware services to lines of business and a second group that provided data warehouse and BI services to the same lines of business. The middleware services and warehousing groups offered conflicting self-promotional opinions that each could better meet the needs than the other.

Getting these folks to change their philosophy was difficult. Top-down management had to dictate that all decision support projects must consider both technologies from day one until the business requirements would dictate the use of one or both.

6: Don't Build it, They Won't Come

In the early data warehousing movement of the mid '90s, many IT organizations embarked on the building of a monolithic enterprise data warehouse. They believed that if they built the mother of all data warehouses, business users would see the value and flock to this new data nirvana. It took a few years, but IT figured out building it would not guarantee that business users would use it.

A disturbing new trend I have seen is organizations falling into a similar trap with integration technologies. "Build it and they will come" is replaced with "Build the bus and they will get on board." Organizations that are tempted should fall back on the basic questions around system implementation: a) what are the business requirements and b) what is the cost justification?

From an architecture perspective, it is important for IT to understand the feasibility of making the connections to all the various data and services, but the business needs to drive the priority of what gets on the bus first. Having a logical understanding of the feasibility and order in which various data and services can be made available to decision support applications puts IT in the trusted adviser role to best govern what ends up in integration and what ends up in lift and shift.

7: Have a Contingency Plan

The business requirements may pose a legitimate need for portions of the data to be near real time using integration technologies, but what happens if the source system has stability issues or cannot support a near real-time connection? This is where IT must have a contingency plan of alternate paths to the data. Although not optimal, a contingency plan may be to have periodic snapshots of the data available for analysis. For example, if a hotel company wants to have near real-time data on their occupancy rates to make on-the-spot promotions to sell out empty space, they may architect a solution integrating through middleware to the reservation system to give the most up-to-date data. But, if the reservation system is having issues or experiences an outage, a backup for the analysis may be to have a nightly snapshot of the occupancy for contingency analysis. The data will have some latency, but it may still aid the business in providing target promotions until the real-time data becomes available.

Evolving technologies and business requirements for decision support, BI and performance management have forced IT to move from traditional high latency warehousing paradigms to a more hybrid approach of combining high and low latency architectures. IT must become the trusted adviser and align the right technologies with the business while paying attention to the long-term architecture impact of a hybrid mode. IT must also be prepared to break down the philosophical and political boundaries that may have separated the two disciplines in the past and concentrate on increasing business value to their stakeholders.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access