Sometimes the difference between a real technology trend and a mere curiosity depends on the organization you're sitting in.
Is "big data" as big a deal in Cincinnati as it is on Sand Hill Road? When we find a cool story, are we writing about the things a few people are talking loudest about, or are we looking at what most are actually prioritizing and doing? What's the best indicator?
Following a emerging tech trend by way of a compelling example can be like grenade fishing. If it's your job to know how many fish live in a lake and what different species are there, one way to find out would be to row around and drop hand grenades; from whatever floats up you can make observations and predictions.
That's a very inaccurate way to sample what's really going on of course. And even the useful and reliable studies conducted by our friends in market research can't undo the skewing of the audience chosen to answer whether it's really excited about something like Hadoop. Ask the folks subscribing to your big data research if they are planning to do something with big data, and they're almost surely going to say "yes."
Though hot topics are always shifting, organizational structure and maturity is often a better indicator that things are really catching on. I was reminded of this by something Colin White from BI Research said on our last DM Radio episode of 2011.
We were talking on that show about the loudest tech trends of the last year – big data among them – which set us to chatting about the stuff of computing nodes and workloads and optimized systems.
Colin pointed out what we were not talking about, which is the organizational impact of big data on the traditional enterprise data warehousing environment, or other new technology approaches surrounding traditional production environments.
While he's excited as anyone about technology trends, there comes a point, Colin said, that unless companies wake up and understand what these new things mean and can do for them in the context of what they are already doing, they are going to risk inertia and getting left behind.
It's not a rush to transition, or the wholesale replacement of anything. It's more about what it takes to equip the organization with the understanding and requisite skills to take advantage of sweet spots in big data – or mobility, or cloud or social computing – as they become practical to use alongside our best production data efforts to date.
A lot of the work in new areas like big data is taking place in organizations and departments outside the BI or EDW environment by different people with skills not yet noted in the HR directory. Pretty much everybody on our radio show agreed on that.
Colin had read a series of blogs on Information Management by our longtime contributor Steve Miller, who'd come up with a bunch of requirements for the new corporate "data scientist." These, Colin found, included business domain subject matter expertise, creativity and communication, statistical and machine learning background, analysis language modeling and mashing data.
You're pretty unlikely to find this mix in any single human unit. And, when you figure that retail is going to have a different flavor and appetite for big data than, say, insurance, and that fiefdoms and demand for big data are going to come out of different places like marketing or claims management, you realize there's not much of a handbook yet for organizations to gather around.
We're not yet sure who to invite to what meeting – and who we're leaving off the list when these projects get started. With apologies to everyone who's doing more (or very advanced examples like Amazon or eBay and the like) most of us are in lab mode.
That's okay, we're early, but as the hype recedes, we'll need to have our stories in order. I'll admit that maturity models (especially advanced ones) are lagging indicators of adoption, and we probably don't need governance for big data like we have for things like master data. But hopefully we'll start to judge adopters and their causes by who they are and what they bring more than we care about how many nodes of data they are running.
Many of us would agree that we'd like to use advanced analytics and other kinds of data sources right now to actually improve what we're doing, but we still need to find the right venues and people to make it happen. As Colin said, we need to bridge the gap between the traditional world and traditional people and the opportunistic work of other people inside the organization. If we don't, we might hurt our cause, or worse, damage both sides.