The conversation on big data is usually directed in terms deriving hidden gems from unstructured data or mining social media for customer sentiment. In enterprise reality, the earliest successes from big data have been from existing volumes of data, though those efforts have been few and far between when it comes to bolstering security, says Crawford.
“We’ve been collecting a lot of data for a long time, and yet we still see attacks – some sophisticated, many more not – that succeed, many with serious consequences,” he says.
This week, IBM launched a real-time monitoring and threat detection platform called Security Intelligence with Big Data. RSA, the security division of EMC, unveiled its Security Analytics platform, with real-time capabilities for IT to track potential risks across multiple data streams in an HTML5 interface.
They’re not the only offerings for big data security, but they’re the biggest announced to date. However, Crawford says that part of the problem with security platforms in general is that they are “predicated on what is already known” on enterprise data.
“This sort of approach limits awareness of what may not yet be known about malicious activity,” says Crawford, who further addressed the vendor issue in a recent blog. “Recognizing the precursors of an exploit, for example, would be highly valuable. Enriching the context of security information to differentiate, say, a legitimate Skype node from a bot that is ‘phoning home’ to malicious command-and-control capability is another example.”
So what has been missing from big data security so far?
“The limitations of legacy platforms that, for example, require data to adhere to a strict schema in order to ingest and use it is an example of why organizations are looking to technologies such as Hadoop and NoSQL environments,” Crawford says. “These have not been trivial undertakings for early adopters – but as products begin to appear that implement these more flexible data management techniques, they will become more available to a wider audience.”
Not merely the problem of vendors, the federal government made it a talking point last week to warn of an “imminent” and crippling security attack to large scale data systems for utilities and other important networks. And the last two months have seen other pronouncements on big data security standards from groups like the Cloud Security Alliance and expectations of discussion at inaugural events on the topic.
“Still to come will be a greater range of analytics that can take advantage of these modern data management techniques to improve data visualization and make findings come alive,” he says. “This will help organizations overcome a key limitation today: it’s hard enough to find quality security expertise, let alone qualified data scientists. Tools must implement this analytic expertise on both fronts to enable organizations to reap the benefits of emerging techniques.”
Justin Kern is senior editor at Information Management and can be reached at justin.kern@sourcemedia.com. Follow him on Twitter at @IMJustinKern.
