A wave by any other name can still lift your boat - or capsize it. The key is balance: lean too far in any direction, and say hello to the undertow.

In the growing field of information management, a sea change feels imminent. Is it a red sky at night or red sky at morning?

James Markarian, CTO for Informatica, recently offered this caution: “You have to maybe reexamine the reasons why you have a data warehouse in the first place.”

Markarian’s counterpart at Teradata, Stephen Brobst, had this to say: “Humans don’t like change. Business process management changes are hard. Technology changes at a much faster rate than humans, so you have to pay attention to the human factor.”

Jeff Bedell, the CTO for MicroStrategy, put it this way: "The next thing is that these systems will just run companies on their own without human interaction.” Though he openly admits: “There are many steps that have to happen before we really start talking about that.”

Those three wise men gathered in Florida last month for the MicroStrategy conference. They had just put out a position paper on what they’re calling “pervasive BI” (business intelligence). Another moniker? Yes, but at least one industry analyst has already given it the seal of approval.

“I actually like the term ‘pervasive BI’ because it has those connotations of not just one thing, like a reporting tool, but a lot of different usage,” commented Mark Madsen, principal of Third Nature. Madsen was asked to interview the three CTOs, who recently marshaled their forces, likely in response to the blockbuster acquisitions of ‘07, when Oracle bought Hyperion, SAP grabbed Business Objects and IBM gobbled up Cognos.

Audio clip of Madsen and Bedell
Audio clip of Madsen and Brobst
Audio clip of Madsen and Markarian
Full interview as wma (stream)
Full mp3 of interview (large file)

Three-way strategic alliances notwithstanding, this concept of pervasive BI certainly poses vast potential and equally stout challenges. On the plus side, spreading insights driven by BI to front-line workers can radically transform an organization, fostering efficiencies that would otherwise be all but impossible. On the down side, mixing analytical queries with operational processes sounds a whole lot like a trip down bad-memory lane, when processor-intensive queries would throw a wrench into the works of mission-critical operational procedures.

Madsen asked the CTOs directly whether their vision didn’t raise again the very issues that led to the creation of data warehouses in the first place: namely, separating operational and analytical procedures to improve performance and mitigate the adverse impact on operational systems that heavy-duty analytical queries can yield. Asked Madsen: “Doesn’t that recreate some of the problems we had that drove us to separate reporting and analysis?”

Responded Brobst: “I think that if we were having this discussion 15 years ago, I would say absolutely yes.” He noted that back then, the data warehouse “was about delivering the KPI [key performance indicator] reports and very traditional kind of reporting mentality; but the industry has evolved significantly since then.”

Just how much has it evolved? Markarian said that today’s leading data warehousing infrastructure is up to the task of mixing intelligence into operational processes - an apparent nod to partner Teradata. “And other parts of the infrastructure like the data integration infrastructure and the business intelligence infrastructure are all up to handling these types of workloads.” Nice nods to Informatica and MicroStrategy.

But let’s get down to brass tacks. Madsen asked Brobst point-blank: what’s the functionality we need to pull this off?

According to Brobst, there are several requisite pieces to pervasive BI. The first is the warehouse’s ability to swallow data on a continuous, near real-time basis. On the other side of the equation, there are such features as event detection and alerting, predictive analytics, plus traditional reporting and dashboards.

All that sounds plausible enough, but there’s one hitch: how do you avoid the kinds of problems that, as Madsen noted, led to the separation of strategy and operations way back when, at the dawn of the warehouse age?

Said Brobst: “You need to be able to support workload prioritization. So I might have a very quick in-and-out query and I need to give that very high priority because it is integral to a decision that is going to be made at the point of interaction with the customer. And that may coexist on the same data warehouse platform with the monster data mining query from hell that is sucking up a lot of resources.”

Multitasking, multithreading, anyone?

Madsen dug deeper: “If we have got a distributed environment like this with continuous loads where we've got ETL [extract, transform and load] running in multiple batches or even streaming data coming in, and we’ve got BI that is using fairly current data and historical data, it sounds like there are some challenges with control points to managing this effectively. So how do you manage or diagnose problems, performance issues so that exercising control in one part of the architecture won't lead to problems in another area?”

Brobst: “You have to be careful not to confuse workload prioritization with capacity planning. You still have to do an appropriate job of capacity planning and make sure that you can handle peaks and valleys,and workload, and support the total workload on the system.”

Markarian, being the data integration guy, offered this: “When you look at modern data integration environments, you're looking at massive multinode implementations, and even forgetting about jumping between technologies; you're going to have different latencies and different throughput on different nodes in your environment. And that is just something that an integration platform has to account for. So you could have potentially infinite data pooling while you're waiting for a particular node to free up at the bottleneck.”

That covers much of the technological challenge, but what of those pesky organizational dynamics? Bedell, as the BI guy in the equation, waxed sociological: “I think that what needs to happen culturally is people need to recognize that to make smarter decisions we need to inject things like predictive analytics into the data stream - not just show the numbers but actually have intelligence as part of the numbers that are coming across to make tactful decisions as well as strategic decisions.”

Translation: there must be a rising tide of awareness within and throughout the value chain (i.e., the enterprise and its partner network), that information really does drive business, that data and the insight drawn from it should play a central, fundamental role in all significant decision-making processes. And there must be budget.

There’s another piece to the puzzle: metadata. Markarian asked rhetorically: “How do you get this picture of what’s going on with your data when you have these fairly elaborate environments?” The answer: “That speaks to metadata, which also speaks to not only administration but also user trust. How do you know exactly what is going on here with your information? What are all of the changes that were made to it, at the time it was transacted to the time it’s showing up on the dashboard? And that too requires the synchronization between all the different element stacks.”

Of course, synchronization works best when it’s tightly coupled, as through application-level integration. There’s another option: via the loose coupling offered by the service-oriented architecture (SOA). So, what’s SOA got to do with it?

Ideally, SOA handles the integration aspect a priori, so to speak. That is, with a comprehensive SOA, in which all messages are properly wrapped with universally understood instructions, we’ll no longer live in a world that requires a posteriori integration, as it were. Rather, integration will occur at every moment of message acceptance: the receptor application will read the bill of materials, then execute the appropriate process, regardless of what that might entail.

From the looks of things, our three CTOs understand the value of SOA: the reference architecture central to their position paper prominently features SOA as the context. Listen to the words of Brobst and you’ll hear plenty of “services” talk, as in the kind of services that comprise the moving parts of an SOA. “Pervasive BI is more than just a dashboard,” he said. “It is really about integrating information into decisioning processes through a broad spectrum of decisioning services within an organization.”

So, if SOA is a threat to the license-heavy information environment these gents put forth, they don’t seem to be sweating that too much. In fact, they make no bones about stressing the importance of stability in their pervasive architecture.

Said Bedell: “You don't see a lot of press about the need for really strong clustering and really strong failover support from the BI platform... And I think you still see too many people talking about future functionality and not about reliability and scalability performance requirements, because ultimately that is what is going to be a key driver for pervasive BI.”

Bedell also offered some specific examples of how pervasive BI might reach the consumer. One example focused on the retail industry: “You’ll carry a garment into a dressing room, and it will pick up the RFID tag. You’ll be presented in the dressing room with information about the garment, care instructions, available colors, et cetera. That’s sort of real-time catalog data."

“But at the same time,” he continued, “you’ll be cross-sold other products. That’s taking market basket analysis into real time, into the dressing room, and telling you: ‘You should also buy these other products.’ I think that’s a cutting-edge use of this technology. Certainly a big shift will be this RFID for retailers specifically. Obviously inventory data is one of the things that’s less accurate today than it will be when RFID becomes mainstream.”

Charting a course to such real-time, consumer-focused decision support obviously requires gumption and stable sea legs. Bedell offered this advice: “I would think that the ultimate way that pervasive BI succeeds is that you have executives such as a COO look at specific processes that need to be improved and figure out how to improve them by injecting intelligence into them. At a very high level, it would be a systematic way of going through every major operational process in a corporation and driving intelligence into each one of those until everything is optimized.”

The COO? Madsen didn’t miss that reference. In BI environments, you often hear of the CEO, or the CIO, or the CTO. But the COO? Noted Madsen: “It’s probably the first time in a BI context that I’ve had somebody refer to the COO, as opposed to the CFO. It does indicate a big, generational shift…”

But again: balance. The lure of new-fangled, real-time decision support may be strong, just as the Sirens drew sailors toward the rocky shore. Just make sure your initiative doesn’t go halfway. Markarian, Brobst and Bedell all stress the importance of reliability and scalability, and not just because they’re promoting their wares.

Bedell: “If the pervasive BI system fails then we're not going to succeed. For it to succeed, it's got to be reliable, and we're trying to draw attention to that.”

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access