Contact Us

What goes wrong with AI agents in the SOC

(and how we handle it)

If you’re putting agentic AI into your SOC, or your MDR provider says they have, this post is the list of questions to ask. It’s based on building the AI layer in CUMULO and as ever it’s our advice, not official guidance.

Why bother at all. CrowdStrike’s 2026 Global Threat Report put average eCrime breakout time at 29 minutes, down from 48 the year before, the fastest ever observed at 27 seconds, and one intrusion went from initial access to exfiltration in 4 minutes. 82% of detections were malware-free, which means attackers logging in with valid credentials and living off the land – the activity that needs reasoning over the event stream rather than signature matching. And the attackers are already using agents. Anthropic reported in November that a state-linked group it tracks as GTG-1002 wrapped Claude Code in an orchestration framework and had the AI carry out an estimated 80-90% of the tactical work across roughly 30 targets, at thousands of requests a second.

Two honest caveats on that last one, because I’d want them if I were reading this. It’s Anthropic’s number about Anthropic’s own product, so treat the precise percentage with the usual caution you’d apply to any vendor reporting on itself. And some of the AI’s claimed successes in that campaign were later assessed as the model overstating what it had achieved – the attackers’ AI hallucinated, which is the first failure mode on this very list. The direction of travel is still clear: agentic tradecraft is real and it’s cheap. You will not keep pace with a human-paced triage queue.

So AI has to do part of the work analysts used to do. Be precise about what part, because this is where most vendor claims fall over. In our architecture the AI is not discovering the facts of a case – it is reasoning over facts the platform has already established deterministically. That is a smaller claim than “AI-first” usually implies, and it is deliberate. The five failure modes below are the price of letting a model touch the work at all, and each one is manageable if you build for it.

 

Five ways agentic AI goes wrong in the SOC and the control for each

Hallucination: Verify every conclusion against the raw events, before action

Prompt injection: Log content is data, never instruction

Acting on a misread: Bounded autonomy, set per action class

Confident wrongness on novel attacks: The council: diverse models, diverse modes. Judgement, never discovery

Silent drift: canaries and continuous validation, on a schedule

1. Hallucination

The AI makes things up when its evidence is incomplete, and it doesn’t know it’s doing it. GTG-1002 above is the live example – even an attacker’s AI overstated its own results. A model missing a piece of evidence fills the gap with something plausible and hands it over with full confidence.

What we do: verification before action. Any conclusion that drives a consequence is checked against the raw events that supposedly support it. The important detail, and the question you should put to any vendor making this claim, is what does the verifying. Ours is deterministic – the AI cites the event IDs behind its conclusion, the platform retrieves those exact events from the SIEM and applies rule-based assertions to confirm they say what the AI claims. It is not a second model giving a second opinion, because two models of the same kind fail the same way and you’d have added a step, not a control. Check fails, the conclusion is held. Evidence incomplete, the AI has to say so rather than reason past it. You also get a complete audit trail out of this, which you will want the day you explain a containment decision to a customer or a regulator.

This is why the SIEM doesn’t go anywhere in an AI-first SOC. It stays the system of truth the AI’s claims are tested against.

2. Prompt injection

Attackers put instructions in fields the AI reads – user agent strings, command lines, filenames, URLs, email subjects, document metadata. Anything carrying attacker-controlled content can carry a payload, and this is not theoretical. CrowdStrike counted adversaries injecting malicious prompts into GenAI tools at more than 90 organisations last year. Researchers at Aim Labs demonstrated the EchoLeak technique against Microsoft 365 Copilot (CVE-2025-32711): a single crafted email, zero clicks, and Copilot could be made to exfiltrate internal data, chaining past Microsoft’s own injection classifier, link redaction and content security policy. It was a proof of concept, patched server-side, with no known exploitation in the wild – but it stands because it defeated multiple layers of purpose-built defence, which is the part that matters.

Be clear-eyed here. The NCSC published a blog in December saying prompt injection may never be totally mitigated the way SQL injection was, because under the hood of an LLM there is no distinction between data and instruction, only the next token. Joint research from OpenAI, Anthropic and Google DeepMind tested twelve published injection defences and bypassed most of them at over 90%. Anyone telling you their architecture is immune is selling you something.

So we reduce the surface and catch the leakage rather than claim a fix. Telemetry reaches the model as structured data with explicit field boundaries, never a free-form text dump. Instruction context and data context are separated at the model interface. And the deterministic verification from section 1 sits behind it – a conclusion that depends on instruction-like content inside a data field gets caught when it’s checked against the raw events and the rules. That is the NCSC’s own advice, near enough: manage the residual risk through design, don’t wait for a silver bullet.

3. Acting on a misread, and the speed problem nobody wants to name

An agent that can isolate hosts and disable accounts can do real damage off bad evidence, whatever the source – hallucination, injection or a wrong analysis. One incorrect containment action against a high-value production system sets your AI programme back years, the ops team won’t trust it again, and frankly why would they!?

So: autonomy per action class, not per agent. We call it bounded autonomy.

  • Enriching an alert – fully autonomous, get on with it
  • Quarantining a single workstation – usually autonomous with notification, depends on your environment
  • Isolating a domain controller or disabling a privileged account – a human approves it, every time

Now the hard part, and if you’ve been paying attention you’ve already spotted it. I opened with a 4-minute breakout. If the high-impact actions are exactly the ones that wait for a human, am I not running my machine-speed defence at human speed for everything that matters? Yes, partly, and that is a deliberate trade rather than an oversight. The reversible 90% – enrichment, correlation, scoping, evidence assembly, drafting the containment proposal – happens at machine speed, so by the time a human is asked to approve isolating a domain controller the decision is one click on a fully worked case, not a cold start. You compress the minutes that don’t need a human so the one decision that does is ready instantly. What you do not do is hand irreversible, high-blast-radius actions to a system that fails in the five ways this post is about. Anyone offering you full machine-speed autonomy on destructive actions is offering you the 3am domain controller story.

Calibrate each class on reversibility, blast radius, and how easily the evidence behind it can be verified. Part 9 covers the gradient properly.

4. Confident wrongness on novel attacks

The hardest one. The AI reasons fluently about an attack it’s never seen and produces an analysis that is structurally wrong but sounds right, with the same confidence as the routine cases. Verification won’t always catch it, because the evidence can narrowly support the wrong conclusion – the model has applied the wrong shape to the situation, not got a fact wrong.

We run what we call the council – the same evidence through more than one model, one prompted to argue the activity is benign, one to argue it’s malicious, one to do nothing but check the evidence chain holds. Where they agree you have confidence. Where they disagree, the disagreement is itself the signal and a human looks.

One technical caveat, because it’s the obvious attack on the idea. If the council is one model wearing three hats, the three opinions share the same blind spots – a novel input that fools the model fools all three prompts, so you’ve mitigated prompting variance, not the model’s actual weakness. To make the council mean anything the members have to be genuinely different models, ideally different families, so their failure modes don’t fully overlap. That is how ours is built. And it has a running cost worth stating plainly: novel and ambiguous attacks are exactly where models disagree, so the council routes your hardest cases to humans and your automation rate dips precisely when something clever is happening. We think that’s the correct place to spend human attention. Across our deployments the council disagrees and escalates on roughly

The AI council

Underneath the council sits the design decision that does most of the work: judgement, never discovery. The platform establishes the facts deterministically from telemetry – which hosts, which identities, what sequence – and the model reasons over established facts rather than generating them. The narrower the gap the model is asked to fill, the less room it has to be confidently wrong about something new. If a vendor can’t tell you where their facts come from before the model sees them, that’s your answer.

Who validates the AI matters as much as the AI

Every control in this post ends the same way: when the AI is uncertain, when the council disagrees, when a high-impact action is on the table, a human decides. That only works if the human is the right human. “Human in the loop” quietly does a lot of lifting in vendor decks, and it usually means a tier-one analyst clicking approve on something they don’t have the experience to second-guess. That is not a control, it’s a rubber stamp with a pulse.

Validating an AI’s reasoning about a possible intrusion is one of the hardest jobs in the SOC, not one of the easiest. You are not checking whether the AI followed a runbook. You are deciding whether a fluent, confident, well-evidenced analysis is actually right – and the whole point of failure mode 4 is that wrong analyses look exactly like right ones. Telling them apart is a job for someone who has worked enough real incidents to know when a clean story is too clean, who recognises the shape of an attack the model has never seen, and who can look at the council’s split decision and understand why it split. That is years of accumulated SOC judgement, and there is no shortcut to it. We have spent well over a decade running detection and response for customers, and if anything the AI has raised the bar on the people doing the validating, not lowered it.

This is the part that should reassure you rather than worry you, and it’s the opposite of how AI in the SOC usually gets sold. The model doesn’t replace the expert. It does the tireless, high-volume, machine-speed work – the enrichment, the correlation, the first-pass reasoning across more data than a person can hold in their head – and then puts a worked case in front of an expert whose time is now spent on judgement instead of grunt work. The leverage is real precisely because the expert is still there at the sharp end. Anyone selling you AI that removes the experienced humans is selling you the confident-wrongness failure mode with nobody left who can spot it.

5. Silent drift

The AI’s judgement slowly gets worse and nothing flags it. Attackers adapt, your estate changes, baselines move, and model updates change behaviour whether the release notes mention it or not. That last one is real and routinely underestimated – a model version bump can shift output format, verbosity and calibration in ways that quietly break a pipeline tuned against the old behaviour. You cannot assume the thing that worked last quarter still works this quarter.

So we run canaries. Known synthetic attack activity seeded into the live pipeline on a schedule, which the AI has to catch. Every canary, every time. Miss one and you’ve found the degradation before an attacker did. Same principle as the heartbeat message you should already have on your syslog feeds. Two operational details people skip: the synthetic activity is tagged at the data layer so analysts aren’t burning real time chasing it, and the tags are stripped from anything the model sees so it can’t learn to recognise a canary and pass the test while failing on the real thing. Get that discipline wrong and the canary is theatre.

AI Canaries

Around the canaries: replay historical incidents and confirm the AI still handles them, red team the AI layer itself rather than just the estate, audit a sample of AI decisions against analyst review, and regression test after every model update. Treat the lot like change control on critical kit, not a deployment checkbox. To put a number on it, our canary catch rate currently sits at

And honestly, this is where the effort goes. The agent is maybe 10% of the work. The canary library, the replay harness, the regression suite and running them forever is the other 90%. If a vendor demos their agent but can’t show you their test harness, you’ve learned what you needed to know.

Be warned: generic AI is not cyber AI

One more, because I keep seeing it. A general purpose chatbot doesn’t know your network, your processes, your change control, your compensating controls or what normal looks like on your estate. Ask it whether an activity is suspicious and you get the average answer for the average network, delivered with total confidence – failure modes 1 and 4 in one. You don’t run the average network, so the advice may simply not apply to you.

Cyber AI has to be grounded in your estate – your telemetry, your baselines, your asset context, your controls. That is the whole reason we built the AI layer inside the platform that holds the estate’s data rather than bolting a chatbot on the side.

The bit that matters

None of this works on its own and none of it is optional. The attackers are running agents at high autonomy and breaking out in under half an hour. The answer is AI doing the reversible, time-critical part of the defence at machine speed, deterministic facts underneath it, an experienced human gating the irreversible actions, and the five controls above treated as operational requirements rather than datasheet features. The AI takes the grunt work off your experts so their judgement goes where it counts. It does not replace the judgement, and the day a vendor tells you it does is the day to end the meeting. It is a lot of engineering. It is a lot cheaper than the alternative.

I’ve made you a promise running through this whole piece – ask your vendor for their numbers, not their adjectives. So here are ours, in one place:

The AI is a component, not the record. The SIEM stays the system of truth.

Sources and further reading

Related Posts

The Year Cyber AI Stopped Being Only About the Model

Authored by: Rob Demain, Founder & CEO For much of the last year, cyber AI was discussed like a model leaderboard. Which model writes the

Read Article

AI SOC vs Traditional SOC: The Real Operational Difference

Author: Rob Demain, Founder & CEO The comparison between an AI SOC and a traditional SOC is often framed as a speed argument. AI is

Read Article