Surfing the Automation Wave with Pervasive BI

BI Review Online, February 28, 2008

Eric Kavanagh

A wave by any other name can still lift your boat - or capsize it. The key is balance: lean too far in any direction, and say hello to the undertow.

In the growing field of information management, a sea change feels imminent. Is it a red sky at night or red sky at morning?

James Markarian, CTO for Informatica, recently offered this caution: “You have to maybe reexamine the reasons why you have a data warehouse in the first place.”

Markarian’s counterpart at Teradata, Stephen Brobst, had this to say: “Humans don’t like change. Business process management changes are hard. Technology changes at a much faster rate than humans, so you have to pay attention to the human factor.”

Jeff Bedell, the CTO for MicroStrategy, put it this way: "The next thing is that these systems will just run companies on their own without human interaction.” Though he openly admits: “There are many steps that have to happen before we really start talking about that.”

Those three wise men gathered in Florida last month for the MicroStrategy conference. They had just put out a position paper on what they’re calling “pervasive BI” (business intelligence). Another moniker? Yes, but at least one industry analyst has already given it the seal of approval.

“I actually like the term ‘pervasive BI’ because it has those connotations of not just one thing, like a reporting tool, but a lot of different usage,” commented Mark Madsen, principal of Third Nature. Madsen was asked to interview the three CTOs, who recently marshaled their forces, likely in response to the blockbuster acquisitions of ‘07, when Oracle bought Hyperion, SAP grabbed Business Objects and IBM gobbled up Cognos.

Audio clip of Madsen and Bedell
Audio clip of Madsen and Brobst
Audio clip of Madsen and Markarian
Full interview as wma (stream)
Full mp3 of interview (large file)

Three-way strategic alliances notwithstanding, this concept of pervasive BI certainly poses vast potential and equally stout challenges. On the plus side, spreading insights driven by BI to front-line workers can radically transform an organization, fostering efficiencies that would otherwise be all but impossible. On the down side, mixing analytical queries with operational processes sounds a whole lot like a trip down bad-memory lane, when processor-intensive queries would throw a wrench into the works of mission-critical operational procedures.

Madsen asked the CTOs directly whether their vision didn’t raise again the very issues that led to the creation of data warehouses in the first place: namely, separating operational and analytical procedures to improve performance and mitigate the adverse impact on operational systems that heavy-duty analytical queries can yield. Asked Madsen: “Doesn’t that recreate some of the problems we had that drove us to separate reporting and analysis?”

Responded Brobst: “I think that if we were having this discussion 15 years ago, I would say absolutely yes.” He noted that back then, the data warehouse “was about delivering the KPI [key performance indicator] reports and very traditional kind of reporting mentality; but the industry has evolved significantly since then.”

Just how much has it evolved? Markarian said that today’s leading data warehousing infrastructure is up to the task of mixing intelligence into operational processes - an apparent nod to partner Teradata. “And other parts of the infrastructure like the data integration infrastructure and the business intelligence infrastructure are all up to handling these types of workloads.” Nice nods to Informatica and MicroStrategy.

But let’s get down to brass tacks. Madsen asked Brobst point-blank: what’s the functionality we need to pull this off?

According to Brobst, there are several requisite pieces to pervasive BI. The first is the warehouse’s ability to swallow data on a continuous, near real-time basis. On the other side of the equation, there are such features as event detection and alerting, predictive analytics, plus traditional reporting and dashboards.

All that sounds plausible enough, but there’s one hitch: how do you avoid the kinds of problems that, as Madsen noted, led to the separation of strategy and operations way back when, at the dawn of the warehouse age?

Said Brobst: “You need to be able to support workload prioritization. So I might have a very quick in-and-out query and I need to give that very high priority because it is integral to a decision that is going to be made at the point of interaction with the customer. And that may coexist on the same data warehouse platform with the monster data mining query from hell that is sucking up a lot of resources.”

Multitasking, multithreading, anyone?

Madsen dug deeper: “If we have got a distributed environment like this with continuous loads where we've got ETL [extract, transform and load] running in multiple batches or even streaming data coming in, and we’ve got BI that is using fairly current data and historical data, it sounds like there are some challenges with control points to managing this effectively. So how do you manage or diagnose problems, performance issues so that exercising control in one part of the architecture won't lead to problems in another area?”

Page 1 of 3.