an Information Management interview with Mark Madsen

Mark Madsen, president of consultancy Third Nature

How is the big and small vendor mix shaking out in BI and data warehousing now?

The obvious change that came out of the buyouts of big BI vendors a few years ago was that SAP and Oracle, for example, decided to gather the whole stack and go for the soup-to-nuts wedding cake for all of everyone's needs. In the old world, a data integration or database vendor didn't do ETL or BI because it would damage their sales. That kept things in stasis for a while. If Oracle had gone out doing BI like Business Objects or MicroStrategy or Cognos, they would have alienated those guys. If an Informatica had come out with a BI product the same thing happens.

After the big BI vendors like Cognos and Business Objects were bought up, it felt like the field of competitors actually grew wider.

Like I said, the different vendors had worked in a nice stratification of BI and parts of BI or ETL. Now the big stack vendors carry the whole thing and the treaty lines have been violated. That's partly why you're seeing all these products like Lyzasoft pop out of the woodwork, turning things around and saying you can embed a column store inside a BI tool, which means you could do things like plumb data without having to build all that infrastructure. You've got a WhereScape saying, 'Don't spend all this design time. Prototype something and if it works, put it in production.' They're violating stack boundaries and layer boundaries, and new guys are saying you can buy a wedding cake, but that's not what we need now.

We see very big vendors demo some new products, but are they content to let those upstarts bake for a while?

The mainstream is their business. They're taking the idea of a data warehouse and BI stack to its logical end point, which is integrating until you have this thing that's soup to nuts and end-to-end. But to me, that's being the Digital Equipment Corp. of the modern age. There are very smart people in those organizations who recognize that, but they can't change the organizational inertia and the feedback loop between the vendors and the customers. They have a lockstep where we offer you product, you consume the product, you tell us the features you need, we rebuild the product. At the same time, people are coming up with alternative mechanisms that can be fascinating, even if they are fundamentally broken in some ways. New things keep coming in on the side that present challenges to the assumptions everybody makes, and it's not the first time the world changes for a lot of people.

Wouldn’t a vendor aiming toward a true utility model to mass-produce capacity and scale most easily want to preintegrate and optimize every layer from the chips up?

In some ways, yes. There will always be people who need that. A small company just needs to do basic reporting, and it's small enough to be operated and managed in a fairly loose fashion. The bigger you are with more complexity in layers, the more other capabilities you are going to need.

That leaves a bunch of loose ends, but I get your point about a fault line.

Well, if you called the data warehouse something like data services and logical models on the front with the metadata for query generation at that layer, then you have the ability to do all kinds of stuff. That's the Internet platform model, right? You build a website, put a bunch of APIs on it and let other people do other things to it. It's like the eBay model, and look at the billions of dollars they make off API traffic.

Won’t the nature of infrastructure keep receding over time, like from a telephone operator to a dial tone, and isn't that what the business guy always wants more of?

That's a pretty good way of looking at it. What I'm saying is that the fracture line is really between the infrastructure that makes data accessible – maybe you store it or virtually access it – but the model sits above it and that's the boundary between the data and the consuming layer. I think the consuming layer stuff in BI is broken because I don't think the BI tool should own metadata anymore. The metadata should be like the dial tone you're talking about. If you can get that out of the BI tools, and of course [BI vendors are] terrified of that and protect it with switching costs. If everybody is using the same metadata, they could unplug tool X and plug in new tool Y tomorrow. Today, that's a nightmare and when you factor in consulting and time to build, there are crazy costs per report to switch from tool A to tool B. The tools want you to do different things, you rebuild all the reports, compare and validate, and you've got to check every single thing.

Video from the interview is available here.