How AI can help narrow the talent gap

The technical skill sets of internal incident response teams are being forced to evolve.

They are transitioning from being predominantly first-alerted to threats and breaches by operational units in their organization to being proactive and self-driven in uncovering and hunting down suspicious events.

The rapid advancement and adoption of centralized log and continuous network event archiving, the construction of “data lakes,” and a new generation of query tools have become the new focus for IR teams.

As a result, they are embracing threat hunting methodologies and employing a new generation of data-mining tools that identify threats in motion and reduce threat discovery and response times from hundreds of days to a few hours.

Senior IR staff – experienced in evidence-gathering and picking through events at a byte-level – have welcomed the new tooling but are increasingly finding themselves being drawn thin on the ground.

As visibility of network and system behaviors increases at an exponential rate, the discovery of anomaly events is happening at a pace far in excess of the ability to deep-dive and pursue a threat to the root cause.

In past years, an event was often “big” before it was noticed, typically by the affected business unit, and the value of throwing in IR resources to understand and mitigate the threat was rarely given second thought.

Now, with hundreds of suspicious events automatically uncovered per hour – and the challenge of detecting events early before they cascade into full-blown breach events – many deep forensic investigative capabilities are being shelved and required less often.

A new problem is developing. How do you retain or justify the retention of deeply experienced IR and forensic specialists? I’m talking about those senior Tier-3 specialists with decades of experience who often come with law enforcement backgrounds.

Forcing these employees to data-mine, hunt for new threats and deal with several dozen early-stage puzzle pieces per day does not leverage their core skills and becomes a disincentive to stay.

Coming to market is a new movement of threat-hunting platforms that harness artificial intelligence (AI), excel at monitoring network traffic in real time, mining the growing lakes of logs and alerts, automatically correlating events and anomalies, and categorizing and labeling attacks in progress.

The previous generation of tools lacked AI and relied heavily on Tier-1 analysts trained in basic data analytics, data mining, and false-positive triaging. They would bundle up evidence files and make a first-pass determination on overall threat severity and decide who gets an incident ticket.

Too many organizations thought they could retrain their experienced Tier-3 teams to perform these Tier-1 threat-hunting tasks. The unfortunate results have included disenfranchisement of technical leaders and atrophy of deeper technical skills.

AI and the automation of the Tier-1 analyst tasks it facilitates are re-addressing this critical workload balance. These new AI-powered platforms are capable of completely replacing the Tier-1 incident analyst and responder roles.

Instead of forcing experienced Tier-3 employees to absorb Tier-1 workloads, new platforms allow the most experienced IR teams to work more closely with the business to address a correctly prioritized list of outlier events.

These outlier events often require a deep understanding of the business, combined with extracting evidence not typically captured in logs or alert events. And, most importantly, they keep highly skilled and knowledgeable experts in the organization motivated and enable them to continually add value.

The application of AI to the cyber security space is already proving invaluable in entry-level roles such as security operations center (SOC) analysts as well as in event triaging, vulnerability scanning, and identity and access management.

The talent pool in these critical areas is dwindling rapidly, and the overall skills shortage has resulted in difficulty hiring, training and retaining people for the first-rung of a professional career in information security.

The use of AI is destined to fill that Tier-1 talent gap. But, as observed in the IR and forensics fields, AI can also enable an organization’s most skilled and experienced staff to focus on solving problems for which they are uniquely suited.

(About the author: Gunter Ollman is chief security officer at Vectra and a member of the ISACA. This post originally appeared on his ISACA blog, which can be viewed here).

For reprint and licensing requests for this article, click here.