Could your choice of metrics be harming your SOC?
Poor metrics can render a well-intentioned security operation centre entirely ineffective.
Emilija Manevska via Getty Images
Security operation centres (SOCs) are a key defence in organisations, where skilled analysts use tools to hunt for attacks in the haystacks of logs. Unsurprisingly, SOCs are usually expensive to run, with costs that include staff, licenses and storage. As both an important defence and a pricey one, organisations are understandably keen to measure and assess SOC performance, whether the SOC is internal or outsourced.
As SOCs typically use ticketing processes to track the handling of detections and incidents, a common approach is to evaluate a SOC's effectiveness by using the same key performance indicators (KPIs) or service level agreements (SLAs) used by other ticket-centric IT departments, such as IT service desks, customer support, or development teams. This will typically include measures like ‘number of tickets processed’ and ‘time taken to close a ticket’. SOC-specific metrics are also included, with a heavy focus on anything that can be:
- easily expressed numerically
- explained to staff who aren't security specialists
Common examples include ‘detection rules written’ or ‘volume of logs collected’.
Unfortunately, many of the most common SOC metrics provide no insight to seniors about the effectiveness of the SOC. In fact, inappropriate metrics can even distract and in the worst case harm a SOC's ability to detect and respond to attacks. The NCSC have seen SOCs with great potential rendered entirely ineffective through poor choice and application of metrics.
How can metrics blunt a SOC?
A consideration when applying metrics is that if they’re used to quantify performance, then staff are incentivised to ‘optimise’ metrics, and this can lead to some perverse outcomes. Let's consider some common SOC metrics, and how they can unintentionally degrade a SOC's ability to detect threats.
Metric 1. Number of tickets processed
When a suspicious pattern in logs triggers an alert rule, it typically produces a ticket for analysts to triage. The analyst assigned to the ticket then has to assess the alert, and make a call whether it might be:
- a real attack requiring escalation into an investigation/incident
or
- a false positive due to a quirk of the alerting logic
In the vast majority of SOCs I've observed, alert logic leads to a lot of false positives. I’ve seen ticket-focussed SOCs where as many as 99% of tickets were being triaged as false positives. This means that an analyst being measured on ‘number of tickets processed’ is incentivised to quickly find a reason to close it as a false positive, rather than to escalate or investigate it.
Metric 2. Time taken to close a ticket
Similar to the above, but the analyst is now also incentivised to click ‘false positive’ as quickly as possible.
Metric 3. Number of detection rules
A subtly dangerous metric as the benefits seem self-evident. It seems logical to presume that the more rules there are to 'detect bad things' will result in more chances to 'detect bad things'.
Unfortunately this is rarely the case.
Such a metric almost always leads to the perverse outcome of ‘alert inflation’; analysts are incentivised to write as many rules as possible, so the metric goes up. However, this leads to false positives as well as ineffective rules. At its worst, I've seen individual rules for individual Indicators of Compromise (IOCs) like an IP address.
Metric 4. Volume of logs collected vs value of logs collected
Effective detection needs good logs, and whilst logs are very useful for incident investigation, logs on their own won't help with detection. I've seen too many SOCs that are ingesting ever-increasing volumes of logs, but those logs often either have limited detection value, or the SOC isn’t using the logs for detection (no relevant alerts, or threat hunts that require those logs).
I visited a SOC where one of their largest log feeds by volume had never been set up correctly, so they only had the first 30 characters of each entry. However, this had never been noticed, so they were not carrying out any meaningful alerting.
Worse still, collecting increasing volumes of logs with limited value generally means the existing logs can be retained for less time (as additional logs will incur additional cost, or take up disk space)
How should you measure a SOC?
There is only one metric that shows a SOC’s efficacy: does it detect (and respond to) attacks in a timely manner?
This is commonly metricised as ‘time to detect’ (TTD), ‘time to respond’ (TTR), or ‘mean time to detect/respond’ (where there is enough data to make a mean useful – for instance where a managed security service provider covers many organisations). However, this can be tricky to measure as an organisation's defence in depth means it should be a very rare event for an attack to make it into the organisation’s environment. As such, organisations may be unable to distinguish between:
- the SOC is effective but there aren’t many successful attacks
- successful attacks are or could happen, but the SOC is missing them
To navigate this ambiguity, organisations can use red teaming and purple teaming to simulate attacks from the types of attackers that are more likely to target the organisation, and both techniques allow assessment and development of a SOC's ability to detect.
The covert nature of red teaming can more accurately mimic a real attack, but purple teaming can often provide better value to a SOC (as the time saved through ‘not being covert’ can be put into greater coverage of attack paths).
Another approach involves breaking down an attack into component (or atomic) stages, typically based on the MITRE ATT&CK framework, and conducting each stage in isolation as a test case that can be assessed as detected and escalated (or not). For example, ‘searching for passwords in a SharePoint’ could be a discrete step in many different attack chains. Some organisations then automate the steps on a regular basis, to test whether they are detected as part of reporting the TTD/TTR.
This can be useful, but risks the perverse outcome of a SOC only detecting a specific/automated way of executing the step. In this example, if the organisation had a rule for people searching SharePoint for ‘password’, then an attacker searching for ‘passw’ might not trigger an alert. Seeing the assessment as something to hone the SOC (as much as test it) can help avoiding this trap.
Whilst TTD/TTR are the only reportable metrics that demonstrate a SOC is working, a SOC manager is likely to want to track a number of other metrics to help them monitor the week-by-week health of their service. These metrics could include things like numbers of tickets, but crucially, those metrics should not be reported outwards (or arguably inwards, to the SOC analysts) lest they drive the wrong activities.
What activities should help timely detection of an attack?
If we accept (or a red / purple team assessment has proven) that a ticket-focussed SOC is less likely to detect attacks, what things can be done to increase the chances the SOC team will detect attacks? Opinions differ around this, but I’d argue that it requires focussing on analysts as threat and tech-aware, enabled experts, rather than a triage layer for tools and alerts. Effective detection needs analysts who:
- understand the systems they're protecting
- understand the threats
- are expert in the tooling they use
- have the data they need to spot abnormal behaviour
- are given the time to put it all together and hunt for threat
Given the above, I’ve found a number of useful approaches:
- 1
Hypothesis-led threat hunting
In my opinion, threat hunting is the most effective activity a SOC can engage in. I generally favour ‘hypothesis-led threat hunting’. In this, an analyst (or team of analysts) creates a hypothesis for an attack that might be happening based on understanding likely attackers and their techniques, or on a tip-off of a specific attacker and their techniques. The analyst then looks to understand the techniques and searches for evidence in logs. In many cases the analyst won't find anything, but the real output of a hypothesis-led threat hunt is the increased understanding of the techniques, and the alerts (or hardening suggestions) that the analyst proposes following the hunt.
- 2
Maximal true positives / minimal false positives
Every false positive harms your SOC by both distracting an analyst, and by incentivising them to expect false positives. SOCs should maintain hard thresholds for false positive rates when evaluating whether a detection rule is suitable for acceptance. Achieving this whilst detecting suspicious behaviour is hard, but is possible through understanding the organisation and its systems, and whittling down known false positive situations.
A crude example is looking for PowerShell execution by anyone who isn't in an IT role, and then working down the few false positives that remain when writing the rule, so that any new PowerShell is an attack (or needs adding to the exceptions). A useful practice, particularly for a SOC transitioning from ticket-focus to analyst-focus, is a regular (fortnightly or monthly) review of any rules that are generating false positives and re-evaluating the detection logic or exceptions.
- 3
Awareness of threats
Analysts need to know what they're hunting for, so metrics around their awareness of threats can be useful. This can be expressed in different ways such as completeness of documentation around a threat actor, or training, reports read and actioned. The MITRE ATT&CK framework is useful for breaking down the problem, and provides tools for seeing which attack steps are used by different attackers, allowing an organisation that understands which attackers may target them to prioritise learning into specific techniques.
- 4
Expertise in tooling
Tracking analyst skill development on the SOC’s tools through training and certifications is a useful exercise. The challenge is doing this whilst making sure analysts remain comfortable working across different tools, rather than getting hyper-focused on their preferred tool.
- 5
Engagement with the wider organisation
Analysts need to know ‘what normal looks like’ so they can detect abnormal activity. This means they need to understand their organisation, its staff, its systems, and how it functions. Metrics can include organisational systems/departments that are documented by the SOC. If a SOC is doing their job well, they will be spotting things that are anomalous (and potentially suspicious) that logs alone don't explain. The analyst will sometimes need to contact the user involved to get to the bottom of it. Such engagement is a great indicator as it suggests the SOC is spotting anomalies, but also the effect on the user is typically to inspire ‘good future security practice’, as long as the analyst is professional and not seeking to blame the user.
IT admins/operations are a department that the SOC will often need to work with for various reasons, and the relationship should be bilateral. Tracking the net promoter score of the SOC by admins (and vice versa) can indicate whether there are issues in the relationship.
- 6
Analyst satisfaction
A SOC job should be rewarding and involve learning about attackers, understanding techniques, applying it to data, and working with people across an organisation. If things are working well then analysts overall satisfaction should be high, whilst poor metrics or culture can often indicate miserable analysts. An overall measure of satisfaction such as net promoter score can help track this.
- 7
Relevant coverage
Whilst log volume can be a misleading indicator, log coverage can be useful where the SOC understands the specific data it would need to detect attacks. Organisations often use the MITRE ATT&CK framework to assess the types of logging coverage needed for attacks of concern. Once the logging needs are understood, tracking the percentage of appropriate assets that are reporting the right logs can help ensure fewer blind spots exist. Organisations need to make sure they’re doing this with knowledge of their environment and the attacks they worry about, as simply tracking MITRE coverage can allow less scrupulous vendors who claim perfect coverage (in highly specific situations) to take advantage.
Summary: bad metrics are worse than no metrics
SOCs are crucial to most organisations' defences and, when done right, are a place where analysts develop and deploy their expertise to hunt and stop attackers. With the wrong metrics, a SOC is ineffective and the job is miserable, with analysts describing themselves as ‘ticket monkeys’ measured on clicking 'false positive' as quickly as possible, whilst being shamed for missing real attacks.
If you’re worried your SOC might be falling into this trap, a red or purple team from a credible vendor will give you proof either way.
In summary, organisations should avoid trying to measure a SOC with metrics focusing on tickets. At best it provides an inaccurate picture, and at worst it actively harms the SOC’s chances of being effective. Organisations instead need to measure whether the SOC actually detects attacks, and how long that takes, whilst driving activities that enable analysts to excel in their work and enjoy their career.
Dave Chismon
NCSC CTO for Architecture
Share and print this article
Written by
CTO for Architecture
