The New Decade of Advanced Analytics: Roll Over Rocket Scientists!

Crafting a truly comprehensive analytics environment is a bit like staring deeply into the night sky. When you try to absorb the billions of celestial objects out there — all of them at different ages and stages in their respective life cycles — you risk driving yourself insane. Your complex field of view contains the deep past, present, and future in one glorious, glowing glimpse.

Increasingly, the complex event processing (CEP) market, as a segment of the analytics arena, suffers from this “all too much” problem. This is not a slap against the technology itself, which is mature, or and the growing list of CEP vendors, who offer many sophisticated solutions. Indeed, many CEP vendors now offer tools for viewing both the deep present, consisting of myriad streams of real-time events, and the deep past, in the form of access to historical information pulled from many data warehouses, marts, and other repositories And some—most notably, IBM with its InfoSphere Streams technology — now support visualization of the deep future, through its ability to apply predictive models to real-time event streams.

The core problem with today’s CEP offerings is that many of them are power tools, not solutions suitable for the mass business market. This same problem confronts established vendors of predictive analytics and data mining (PA/DM) tools, whose core user base are statisticians, mathematicians, and other highly educated analytics professionals. No one denies that traditional CEP and PA/DM tools are the analytical bedrock of mission-critical applications in diverse industries. But I challenge you to point to a single case study where they are used directly by the CEO, senior executives, or any other information worker, rather than indirectly through being embedded in some custom application.

Forrester regards CEP and PA/DM—the deep present and future, respectively—as key to the evolution of business intelligence (BI). However, these technologies can’t realize this potential until they break out of their golden ghettoes. BI vendors know this, and are focusing in 2010 on integrating both technologies into their solution portfolios while providing user-friendly visualization and development tools that submerge the complexities and accelerate time to insight.

In this race, the vendor that has integrated BI, CEP, and PA/DM most effectively for the mass market is TIBCO Spotfire. The vendor offers user-friendly, powerful, interactive visualization in its BI offering, which integrates out of the box with the PA/DM technology that it acquired from Insightful and the CEP middleware in its parent’s established portfolio. Its solutions enable customers to discover new and actionable insights in information at rest or in motion, while proactively detecting and responding to events. TIBCO positions its Spotfire tools as suited both to traditional data mining specialists and to the legions of non-technical business analysts. Just as important, it can push predictive models to CEP, data warehouses, business process platforms, and other application platforms.

Let’s not kid ourselves, though. CEP and PA/DM remain power tools at heart for the foreseeable future. Over the coming 3-5 years, they will take up residence in many vendors’ BI solution stacks, but still be employed primarily by power users, rather than by casual users, for whom canned historical reports are all they need. And most power users will primarily require lightweight CEP and PA/DM features integrated into their BI visualization layers.

In other words, CEP and PA/DM will still primarily be “rocket scientist” territory. But here’s an important note about this new decade: space tourism will flourish and you too may someday be a casual astronaut.

Thank Sir Richard Branson for that.

James Kobielus also blogs at blogs.forrester.com/business_process/.