SOC Metrics In Cyber Marketing

Have you noticed how SOCs are presented as inherently dysfunctional by cybersecurity vendors? If you’ve been to LinkedIn, I’m sure you’ve seen some version of “SOC analysts are facing 10 000 alerts per day”, “65% of analysts confess to ignoring an alert that later turned out to be a security incident”, “Over 50% of all alerts SOCs are facing turn out to be False Positives”, “71% of SOC analysts report burnout“ etc.

And there is always a report to support those numbers. Today I write specifically about those reports and explaining why I think we should not pay too much attention to them.

I’m also venting a little.

In an industry built on skepticism, it's surprising how easy we trust in reports.

A prime example is the widespread coverage of the ISC2 Cybersecurity Workforce Report. It seems to be taken as gospel without consideration that one of the largest certification bodies in cybersecurity might have a vested interest in the public believing there’s a cybersecurity skill shortage.

SOC workload reports are no different. If anything, they’re worse. I’m convinced that most SOC related reports (all I’ve seen, can’t claim to have seen them all) are misleading readers on purpose.

Here’s exactly what is wrong with them

Probably the most used SOC workload number is the alert volume. No wonder, it speaks to most people better than MTT-X, TP / FP rate or detection coverage. It’s an easy message to digest and an easy, scary number to wave around.

What are those numbers?

Those are just 10 random reportings on SOC alert numbers. As you can see, the difference between those numbers can be massive. On one side we have report by Anvilogic with a reasonable looking number of ~286 alerts / day. On another, there’s Fortinet with a claim of 150,000 alerts / day, which is insane.

Just working with this list we could push any narrative we chose.

You’re probably wondering how we can have a ~525x spread for the question of - what is an average daily SOC alert volume. I don’t know, but I have a hunch. Hear me out.

Most SIEM data tables are immutable - meaning you can’t change records that are already there, you can only add more. In practice, every change to an alert - tagging, commenting, assignment, change in status etc. creates a new record in the table that holds all security alerts.

Now, if we don’t account for those duplicate values and simply query for all records in that table, we will end up with 3-4x the amount of alerts than we actually have.

If you enable your DLP alerting today, tomorrow you'll see 10,000 new DLP alerts in your SIEM. It is what it is. Threat detection tools can generate an absolute mountain of alerts, and the job of SOC teams is to tune that number down to a manageable level.

Now, that means your SOC can receive 10,000 alerts every day, but it doesn't mean your analysts need to go through all of them. Likely 90% of that volume is tuned out, autoclosed, aggregated, you name it.

Which means the question "how many alerts do you receive every day" should really be "how many alerts do your analysts actually have to investigate every day." I can't prove it, but I have a strong suspicion that many of those inflated numbers come from the fact that some SOCs share values prior to tuning.

In-house SOC analysts and MSSP analysts work differently. In-house analysts typically handle an alert end-to-end (or at least a larger portion of it) and have other tasks as well, like monitoring for user-submitted tickets. MSSP SOCs often go through a significantly higher alert volume in a day. That's because instead of fully investigating alerts, they're often expected to escalate them to clients for further investigation, and they typically don't have other tasks outside security monitoring.

In-house SOCs work for themselves; MSSP teams are typically larger and work for multiple clients. Now, based on what I know about SOC work, I'd say an in-house SOC analyst can handle about 20-30 alerts per day. An MSSP analyst probably handles close to double that.

So the daily alert volume of a 100-analyst MSSP SOC with 20 clients can be 5,000. At the same time, an in-house SOC with 4 analysts might see closer to 100 a day. Both numbers are realistic, but if I mix them together and say that a SOC on average handles 2,550 alerts every day, we're being misleading.

Similar to in-house vs MSSP - bigger teams are typically dealing with more alerts. We can’t compare the overall daily volume between a team of 5 and a team of 30, regardless of in-house or MSSP. This is a no brainer, but also rarely accounted for.

All those problems exist before we even consider the possibility that those SOC alert volumes are fabricated. Because, why not? Nearly every one of those reports is paid for and then used as a marketing prop. It's not a stretch to assume the numbers are tampered with in at least some cases. The incentive is there, and we never see the raw data behind any of those surveys, just the conclusions.

So, what do we do? Are SOC workload reports just useless? I’d argue - yes. But, we can improve them.

That way we have a more honest data point.

If this way of thinking about SecOps resonates with you, I'll be sending newsletters out weekly. The best way to follow along is to subscribe to the SecOpsPOV newsletter below. No spam or vendor pitches. Just more of this.