In my last column, I wrote about the need to “understand the normal” – that is, when faced with a complex security incident, to know whether each step in the incident is normal for that user. It sounds obvious; of course it’s valuable to know whether someone is acting in an unusual manner. However, creating context is much more difficult in practice.

Data volumes are growing quickly, often so quickly that it’s not possible to store enough historical data to support an investigation. At the same time, firms have replaced employees with temp workers, contractors, and outsourcers, many of whom turn over on a regular basis. This flux makes it harder to identify actual users, let alone each user’s normal behavior. Simply put, the pace of change and growth prevents most large organizations from understanding normal and abnormal behavior.

This is because the existing model for security analytics and intelligence is flawed. Firms rely primarily on people to analyze data and make judgements, but the volume of network activity overwhelms human analysts’ ability to make sense of that activity. The option of throwing more bodies at the problem disappeared years ago; most companies don’t have the budget to hire all of the expertise they’d require, even if enough trained analysts existed (they don’t). The obvious next option has been automation, i.e. let the machine do it.

Automation has been applied in various forms for many years. Scripts to automate data collection worked moderately well when there wasn’t too much data to collect and it was in a common format. Signatures worked moderately well for certain types of attacks, as long as the techniques didn’t change often.

More recently, event correlation worked reasonably well for well-defined, network-based attacks, such as the oldie but goodie “Fred is logged in from home over the VPN, but also just badged into the building. That’s not right!” In short, security analysts could rely on machines (automation) to help when tasks were simple and static, and you didn’t need smart machines. Does anyone think that we are living in that environment today?

Fortunately, though the security environment has gotten more difficult, machines have become much smarter. Machine learning algorithms have progressed significantly in the past few years, and can perform tasks that the leading edge security products of 2011 could only dream of. So, if the machines are more capable, how should we view them?

In a recent presentation, Jerry Chen of Greylock presented two ways to look at machine automation: Iron Man or Terminator. In the Iron Man scenario, intelligent technology turns a human into a superhuman. Iron Man (Tony Stark plus his machines) is far more capable and powerful together than apart. In contrast, the Terminator scenario is awful: humans live in misery thanks to smart machines.

Putting the pieces together, the only way that IT organizations can cope with the volume of security data on their networks today is by augmenting human intelligence with machine capabilities. Using the example above i.e. is this person’s behavior normal, it is very common for a SOC analyst to spend a week or more gathering the necessary data, preparing it, and analyzing it, simply to answer that question.

In contrast, a good behavioral analytics system can provide the same answer in seconds, with less likelihood of error. These systems are constantly ingesting new activity data, updating baselines, stitching individual activities together into timelines, and analyzing these for risk. The level of automation and accuracy is surprising, even for someone who has worked in security for many years.

This was not true two years ago but is true today. Given this level of power, plus a shortage of hirable expertise, why would any organization not take advantage? It’s time to stop fearing the machines, and to use them to become Iron Man.

(About the author: Nir Polak is chief executive officer and cofounder at Exabeam)

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access