Orlando – A funny thing happened here at TDWI's annual fall conference – and it was intentional.
Amid the technical classes the conference is known for, the theme of the show was the impact and implications of “Big Data” as we’ve ALL come accustomed (as if you're not already besieged by the term).
So, around the case studies and technical discussions, a pair of high end consulting types decided to get NOT serious in a session over the phenomenon.
Mark Madsen of Third Nature and Marc Demarest of Noumenal were the players squared off in a mock debate. In one corner, Demarest, sporting a “Trust Tech” t-shirt and playing the gung-ho CIO; and in the other corner, Madsen, with “Fear Change” splayed on his chest, playing the rational contrarian to the addictions of big data tech.
What follows is tongue in cheek (but I can easily imagine might be heard in any given corporate hallway). And remember, these two were among those first anointed in dimensional schema design by Ralph Kimball himself.
Demarest (the optimist): “The big data revolution is big, it’s real, it’s now, it’s not optional. It destabilizes our current notions about just about everything and five years from now, things will be mostly different.”
Madsen (the contrarian): “[Big data] is a sideshow to a circus. … We are in the middle of a hype cycle … and we have a business to attend to and so all this stuff that’s going on will get baked into the things we’re already doing. My colleague (Demarest) is completely divorced from mainstream America.”
Demarest: “Everywhere you look these days … companies that are successful in increasing and growing their share do so not by producing better products, but by wrapping their products and services in high value add information and micro-targeting increasingly differentiated constituencies ... McKinsey and Booz Allen Hamilton are telling our CEOs this every day …”
Madsen: “We’ve been treating data as an asset for 20 years. But it’s not a factor of production for most people unless you are a startup or something like that or possibly in the financial sector where you’re not producing discrete individual goods. Otherwise it’s just a piece of helping you manage your business. What we’re dealing with here, things like this McKinsey report or this Booz Allen Hamilton report is that these guys just woke up and smelled money the other day.”
Demarest: "We are all going to be inundated (with data). What’s going on … is that this phony world we created in the 1970s populated with computer systems that were supposed to give us a view of what was going on in the business is now being linked up in every way imaginable to the chaotic mile-a-minute real world, the net effect of which is tons and tons and tons of data of all sorts. Velocity, variety and volume is real, and it’s real for everybody.”
Madsen: "It is true, there is a lot more data now … We’ve added some new sources but a lot of this astronomical growth .. is a little misleading because we’re getting …just small insights from vast amounts of data. That Web data, monitoring Twitter, if you’re not in a consumer facing business you’re probably not concerned with that sort of stuff. Does a GE refrigerator really need a Facebook fan page?”
Funny stuff, like I said. Playing devil’s advocates, what Madsen and Demarest demonstrated with humor is that you can sit on either side of the new technology fence and still be right at least part of the time. That’s where we are right now with the craze of Big Data. A lot of interest and potential in Hadoop, new architectures and new approaches. It’s good that data managers are excited again, Madsen said.
You also see a lot of opportunistic marketing, and some people who are frankly drunk on big data. Said Demarest, “this is either the biggest con or the biggest sea change in our personal odysseys since the early days.”
It’s probably somewhere in the middle and a reason to pause as well as be excited. What do these two think will really happen with big data?
Madsen: “Data is a factor of production, mainly for digital companies, but there is a lot of information out there to do things differently ... Companies are taking these technologies and using them in novel ways and changing the way a retailer can configure their business around a customer instead of around the supply chain like WalMart ... but you have to pick and choose which opportunities are good for existing technology and which make sense for the new world.”
Demarest: “Reality of big data? There is a lot of hype, it remains somewhat mysterious to me what people are going to do with the full Twitter fire hose. … It’s reasonable for any firm to assume that over the next five years the volume of data that they have under management … is going to grow by a factor of two to five, but we’re not looking at a scalar transformation from terabytes to petabytes. For most firms, I think we are looking at increasingly complicated analytics and the data to support those.”
Madsen: “I agree with that. The computation over the data is a bigger part of the problem than the volume of the data itself.”
Demarest: “I’m a big believer that the heart of analytics is computation intensive and algorithmic. We’ve heard a couple of speakers say this today, that we’re probably over visualized and under algorithmic-ed or whatever that word would be.”
Amid other interesting points, Madsen and Demarest agreed that a lot of the world has changed, that BI is a more mature practice, and most of the low-hanging fruit is gone. The next thing will arrive, Madsen figures, but it will take more work and effort.
Demarest says fear not, that the basic architecture we have already for data is okay, it’s simply insufficient. We’ll keep fundamental architectures, it will be more complex and expensive to manage, but it is not something we need to dispose of and replace.
So, while minding the hype, remember, Demarest said, that it took a long time to get to where we are already. “We are not fundamentally challenged, we know what to do.”
The good news (and the bad) seems to be that there are many paths and opportunities to pursue. Whatever we wind up calling it, we’re going to be at this a long time.