Big Data As You Go

While making the rounds of news and reports this week I came across a note from Gartner Research VP David Newman, who was comparing the phenomenon of big data to what he called “open data,” a term to describe transparent availability of data streams and services through public sources and commercial services.

Newman was going over ground he’ll cover at Gartner’s flagship Symposium and IT Expo in Orlando in October and it’s not a new topic for him or others, but I think it’s a very underestimated one. Part of what Newman talks about is an economy based on data services, connecting the assets of governments, open projects and proprietary data streams as free or paid services.  

It’s a topic that touches home for Internet commerce, the idea that data itself is destined to become its own currency, based on streams of connectivity and application programming interfaces (APIs) acting as a gateway or virtual point of sale system for conducting data commerce. Here at Information Management we spent a long cover story on the topic almost three years ago, maybe a little too early to the game. Even back then, an independent analyst named John Musser said “for every cloud platform, APIs are the glue of SaaS.” 

In fact, API traffic was outstripping browser traffic years ago and repurposing proprietary information to stream as raw product was part of that picture. That’s been the case for information resellers like ADP, D&B and Hoovers, or TransUnion which years ago got out of the business of building websites for partners and got into the business of providing data for partners to use as they chose.

Though we’d found some of the same examples Gartner did, Newman has a different and even broader view than we took, extending to examples of data.gov projects in the U.S. and U.K. as examples of self-service data access. He also talks in terms of linked data and social data as platforms of data commerce and third parties acting as intermediaries or data brokers.

“More government agencies are opening their data to the public Web to improve transparency, and more commercial organizations are using open data and the API as an alternative way of generating revenue,” Newman says. “New business models are emerging and it’s exciting for a data geek like me.”

The point, he told me, is that “big data makes you smarter but open data makes you richer.” The question for organizations is how they will use their own data alongside public and paid information to make money or get closer to customers.

It’s business context to the technicalities of data management that cut through hype, says Newman, who’s an experienced enterprise architect. “Really I’m trying to help other enterprise architects understand how to get to value because unfortunately we’re in an era where everybody is looking for a silver bullet. People hear big data and they think, ‘I have to have one of those.’”

It’s a data economy that’s presently full of challenges, in trust, complexity, cost and how to govern, walls are slowly coming down, Newman says. “I wasn’t the first to say this, but you could make a case that in ways this idea of open data it’s like an iTunes for raw data.” Financial and proprietary data aside, you can look no further than your dusty CD collection to imagine that might become the case in organizations, if it's not so already.