If 2007 was the year of business intelligence (BI) consolidation, with the acquisitions of Hyperion, Business Objects and Cognos by Oracle, SAP and IBM, respectively, then 2008 will be - what? Conventional wisdom suggests this will be a year for digestion - neither the year of the snake nor the pig, rather the year of the pig in the python. But conventional wisdom will be wrong in three ways.

Data warehousing architecture will be rationalized. Granted, the need to process the acquisitions will create some coordination costs and distractions, but it will still unfold, like a caterpillar into a butterfly, into platform-based BI. The acquisitions will actually create opportunities for technology innovation based on the interaction and integration of architectural layers. The front end of the BI system will become more scalable and rich in actionable information, while the back end, the persisting data store itself, becomes more accessible, manageable and useable. Flexible architecture is required to accommodate future business requirements that cannot be foreseen by the conventional wisdom, and the acquisitions will enable the creation of a critical mass of architecture to integrate. This work in progress will make significant headway.

The acquisitions will reduce vendor risk. The end game for best-of-breed, standalone BI tools is now in sight and provides closure for the future of the respective platforms. End-user enterprises and buyers know which vendors take good care of their acquisitions by designing and implementing strategies and roadmaps, which ones merely increase the price of maintenance and which stall in analysis paralysis.

“Data warehousing made simple” becomes even more simple. Consolidation will bring closer to realization the possibility of data warehousing made simple, not as a dream, but as a realistic opportunity. This will occur by means of front-end BI technology integrated with the middle layer of data integration for complex heterogeneous data. In turn, this will benefit the operational business information persisted at the back-end data or information warehouse, where third-generation systems will close the loop back to the front end, optimizing operational processes. In the short term, however, BI platform wars between vendors will be added to data warehousing religious wars and appliance wars. However, these will be lessened by the requirement imposed by end-user enterprises on their best vendors to cooperate for the benefit of clients in the morning, even if they compete in the afternoon.

The Convergence of BI and Business ProcessManagement (BPM) Will Accelerate

The convergence of BI and BPM will accelerate as well as reduce uncertainty and risk about the future of the respective BI platforms, making it clear that BI has never merely meant front-end tools. Tools that just run queries and generate reports have become a commodity. Going forward, the ability to deliver insights directly into a business process, along with pattern matching, advanced analytics and nonobvious relationship analysis will be the domains to which incremental value will migrate.

To manage the business, you have to measure it. To measure it, you have to understand it. To understand it, you have to model it. Modeling the business processes will be the critical path to authentic performance (and process) optimization. This will be confrontational to corporate staff members who are comfortable in their functional silos and turf, but it will be liberating to professionals on both business and IT sides of the house who are committed to delivering value to the whole enterprise by way of its customers (or mission in the nonprofit sector).

At the front end, the key technologies include visualization, dashboard/scorecards, predictive analytics and anything that furthers end-user self-service. At the back end, data warehousing will move beyond data integration - in some cases without having fully mastered it - to business and technology integration. In many cases, this will be driven by the requirement to leverage master data management across operational and analytic systems. Business analysts understand what the customer, product, market, etc. dimensions of master data are and will “coach” their IT colleagues if the latter lag behind. In the middle layer, advances in metadata and the usability of business processing (and performance) modeling will enable the different layers of architecture to work with one another while providing key service in data profiling (auditing), data quality and intelligent information integration.

The End of the Beginning for the Data Warehousing Appliance

The special-purpose, prescriptively defined, shared-nothing database - now commonly known as the data warehousing appliance - will continue to have market traction, with discounting reaching the point of no return. In the short term, this will result in the proliferation of singleton data marts hastily implemented to address a tactical business pain, scratch a technology itch, or demonstrate to a large vendor that it is not the only game in town. However, the intensifying competition will confirm appliance and appliance-like entries from the “big guys”- many already in production in client installations - resulting in the early innovators being leapfrogged by a second wave of major power players.

In addition to innovative technology, the larger vendors will provide significantly reduced business risk, superior maintenance and a coherent roadmap for leveraging the appliance approach for broader enterprise data warehouses, all for a modest premium. Thus, for data warehousing appliances, this is the end of the beginning. The emergence of the data warehousing appliance has been validated by the market notwithstanding that the public filing by the original data warehousing appliance vendor indicates it has never had a profitable quarter.1 Startups have primed the pump, gotten traction, turned some prospects into customers and demonstrated that the idea of an appliance is capable of changing the economics of data warehousing in favor of cost-sensitive buyers. And it has done so in favor of buyers across all platforms, even those that are proprietary.

The end-of-the-beginning market means the start of the mainstream middle market where the technology breaks out into the general purpose data management market. At the same time, it is the beginning of the end for special-purpose, proprietary data warehousing systems, which, henceforth, are renamed “legacy appliances.” Thanks to database innovations in standard relational technology (such as dynamic optimizers, SQL predicate pushdown and transformation, and extreme workload management), going forward, enterprises will need only one kind of database to perform both transactional and BI processes, though it will be common to continue to implement separate instances due to operational efficiency.

Real-Time and Near Real-Time Update of the Data Warehouse Advances

Daily updates will continue to be the predominant approach in the majority of enterprises. Daily updates are so common that they will still occur in about 80 percent of the installations even if the expressed expectations of survey respondents occur as anticipated. Even so, data warehousing latency continues to shrink as demonstrated by the Data Warehousing Satisfaction Survey.2 Clearly, daily update and loading of the data warehouse is the state of the art today, with 88 percent of enterprises responding that is what they do (see Figure 1). As indicated in Figure 2, daily, weekly and monthly updates are anticipated to decline by about 11 percent, 11 percent and 17 percent, respectively, with the slack taken up by near real-time and real-time updates.

Thus, an emerging trend toward reduced latency is becoming clearer. Near real-time updating of the data warehouse and multiple daily updates will take off and explode if enterprises behave as expected. From a relatively modest base, the update frequency within these “dynamic” categories grows by 117 and 200 percent, respectively. Thus, I conclude that dynamic data warehousing is reaching an inflection point in the market. It is reaching take-off speed. Those enterprises still operating legacy data warehouses that are trying to use a trickle utility to manage a fire hose of data will increasingly be at a disadvantage.

Social networking will continue to seek a revenue model, while generating reams of data. The most disruptive technologies of social networking, data mashups and wikis, including open source versions of these, will be limited in their immediate impact to community building and collaboration, where they will make a significant contribution. The technology is out front, but an overabundance of risk aversion and a shortage of imagination will inhibit adoption outside of the educational, help desk and media industries. Still, it is a start. Enterprises that succeed in managing the risk and filtering out the predators and scammers will reap the rewards of viral marketing, customer knowledge and mind share while learning to manage significantly more data than ever before.


Envisioning the future is always a challenging undertaking. There are at least two ways of going about it. You can take two data points in the present (or past) and draw a line through them into the future. This is useful in saying how the future will be like the past, as is often the case. Data warehouses will become bigger, faster, better. This is supported by example: as reported in the Washington Post, the FBI will invest $1 billion in a biometrics database of faces, finger print, iris scans and related physical data.3

However, another approach to prediction is available, one often used at enterprises with elaborate laboratory systems, innovation agendas and commitments to knowing the future by creating it is a possibility. Stand in the future and look back toward the present, and then say what intentions and commitments are needed to create such a future. With that in mind, three recommendations are proposed:

Invest in the front-end BI platform. Follow the fundamental design principle that business value migrates in the direction of the user interface. Experience shows that no matter how much upstream data collection, information transformation and intelligent integration occurs, the payoff in the experience of the client occurs at the user interface. Whether business analyst, electronic file clerk, top executive, system administrator or power user, the “ah-ha!” moment occurs at the user interface.4 Information delivery provides a last opportunity to synchronize the experience of the user with the business process that is unfolding in the information supply chain system. Accomplish this by means of a BI platform roadmap that allows for integration with business process (and performance) management. This is a basic principle in thinking about system architecture and the design of a usable software interface.

Invest in metadata in the middle layer. What do data integration, data quality (and cleansing) and data profiling (and auditing) have in common? Nothing, unless they are linked together by metadata and the intelligent information integration that metadata provides. Where practical and cost-effective, such integration can best be accomplished by provisioning a platform that offers these capabilities by means of a data hub. That is different than a data warehouse because it does not necessarily persist the data as it transforms it and provisions the relevant data warehouse, operational data store or the occasional tactical data mart.

Invest in the back end. Avoid data warehousing religious wars. Get ready for dynamic data warehousing and operational integration of business information. Attack costs by means of autonomic computing, virtualization and intelligent workload management. Understand that pure-play appliances are best managed by a process suitable for dependent data marts but that the dynamics of the market are fundamentally being transformed by the entry of large vendors. Competition will continue to be intense at the high end of the data warehousing market, catalyzed by the appliance trend. However, the changing economics of data warehousing (as driven by Linux, open source data warehousing platforms and more powerful computing chips) will accrue to the advantage of end-user enterprises that will gain more data warehousing for the dollar. The “more” consists of built-in analytics that leverage the back end to deliver information in easy-to-use packages to the front end, the ability to roll up unstructured data into the data warehouse (native XML), reducing latency by means of high-speed inbound processing and information delivery, and extending intelligent information integration to the applications dependent on this enterprise infrastructure.


  1. S-1 Filing. Netezza Corp., March 22, 2007.
  2. Lou Agosta and Kevin Modreski. “Data Warehousing Satisfaction Survey, Part 3: A Single Fact is Worth a Thousand Opinions.” DM Review Special Report, October 2007.
  3. Ellen Nakashima. “FBI Prepares Vast Database of Biometrics: $1 Billion Project to Include Images of Irises and Faces.” Washington Post, December 22, 2007.
  4. Keith Gile. “Profiling the Analytic End User for Business Intelligence.” Forrester Research, June 26, 2003.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access