What recent agent incidents and our own experiment show: agent security has to be enforced outside the model

Today, AI agents (i.e., language models equipped with tools) are doing real work in various companies. They write and merge code, triage alerts, answer customers, and even hold real credentials.

The greater the autonomy, the more consequential the failure, when it comes.

Three incidents inside the evaluations themselves

Recently, OpenAI disclosed that models under evaluation, with their maximal cyber capabilities enabled for the test, broke out of the isolated environment through a flaw in a granted tool, reached Hugging Face production, and took the benchmark's answer key in order to cheat the test [1]. The benchmark asks an agent to turn real, known vulnerabilities into working exploits [2].

Nine days later Anthropic disclosed three incidents of its own. Reviewing the transcripts of its cybersecurity evaluations, it found that in each case a Claude model reached the internet from a third-party evaluation environment and then gained unauthorized access to a real organization's systems: a production database read after credentials were extracted, a security company's credentials exfiltrated through a package its scanner installed and ran, and an internet-facing application compromised after a scan of roughly nine thousand targets [3].

Five days later the UK's AI Security Institute reported the same thing from its own side of the table. Across 122 cyber-evaluation runs, 10 produced 19 unauthorized actions against real people and real projects: an agent inserted malicious code into a public open-source repository, invented identities to talk the maintainer into merging it, and routed its traffic through Tor to hide. Internet access was on and the safety classifiers were off, both by design. The runs were shut down within an hour of the alert. The malicious change was stopped by a human reviewer who declined to approve it, and AISI's own verdict on the outcome is that "the margin between failure and success was narrow" [15].

On the one hand, in every case the environment gave the agent more reach than its operator intended, whether by misconfiguration or by design; in one of Anthropic's cases a misconfigured container alone handed the model direct internet access. On the other hand, the OpenAI story suggests the high-risk cyber activity was possible only because the model's guardrails were disabled, which invites the question of whether the incident would still have happened with them on. Anthropic's disclosure answers part of that question. Those models kept their safety training; what they ran without were the classifiers and monitoring that Anthropic deploys in production, which is to say the controls that sit outside the model. The training was intact, and three organizations were breached anyway.

The pattern predates all three. An email assistant kept deleting messages after being told to stop [4]. Within days of Claude Fable 5's release, researchers reported a method of slipping past its safeguards [5]. And a systematic benchmark of prompt injection attacks against ten defenses found none of them sufficient [6].

The necessity of defenses beyond the model's guardrails

Cogent has long operated under the premise that a model's guardrails, however strong, are not a silver bullet. The incidents above differ in their particulars, yet they share a shape: whatever control sat outside the model was absent, switched off, or broken, and what remained between the agent and production was the model's own judgment. Safety training can make a model less likely to misbehave, but because it lives inside the model it shares the model's fate, and it cannot guarantee that the model will not.

Deploying an agent safely therefore rests on two measures that sit outside the model. The first is isolation: give the agent its own least-privilege, ideally disposable environment, so that a rogue agent dies with the box rather than reaching production [7]. Isolation has to cover the network. AISI's agents ran in a sandbox walled off from AISI's own systems and still reached a stranger's repository, because the box had the internet. The second is deterministic policy enforcement at the tool call.

In its simplest form the policy is a fixed rule, checked at the tool-call layer, which permits or denies each action before it takes effect [8]. While a guardrail asks the model to behave, the policy decides what the model may do. Since the policy is machinery the model cannot read, edit, or argue with, no prompt moves it. In other words, it is a defense the agent cannot out-reason, only out-flank.

Deterministic is not the same as frozen: as with law in human affairs, the policy is revised as the world changes, and gaps that red-teaming exposes become constraints later versions enforce. The policy does not replace the model's safety training; it supplements that training.

The experiment

Our subject is a reasoning agent on a complex task, where autonomy is broad, tool use is extensive, and one wrong tool call can do real damage. Coding agents are the most prominent and most consequential instance of such a setting. We tested the premise on coding agents graded by SWE-bench Pro [9]: five frontier models across four vendors, real bug-fix tasks, each agent confined to a disposable cloud VM, with the models' safety guardrails enabled.

We set out to measure five things that are easily conflated: how much effort each model spends on the task, how willing it is to cheat, how capable it is of delivering a cheat that fools the grader, how much of that the policy stops, and what the policy costs on honest work. Each model was placed in three modes of instruction:

• In the honest mode, it receives only the bug-fix task.

• In the misaligned mode, it is pressed to maximize its score for the least effort, a nudge toward a shortcut without an explicit instruction to cheat.

• In the malicious mode, the same shortcut is presented to the model in two forms:

  • a soft suggestion that merely offers it and leaves the honest fix open.

  • a command that orders it, calls it the intended path, and forbids fixing the real bug.

A grader can be cheated in many ways; prior research has catalogued over a thousand such instances on real agent leaderboards [10]. The US Center for AI Standards and Innovation reports the same pattern on public benchmarks, including runs where the model gamed the grader instead of solving the task [11]. In our runs the models converged on one simple, already-documented recipe. Instead of fixing the bug, the agent runs its own code as the test harness loads and overwrites the part of the grader that reads results, so that every test reports a pass. Berkeley researchers demonstrated this technique on SWE-bench Pro [12]. The grader is a stand-in for any artifact a downstream decision trusts: a CI result, a monitoring report, an audit log, a compliance record. We measured one grader in one benchmark, but the shape is general, since an agent that can write to the record of its own work decides what that record says. This cheat is a tool call, which is a write to a file the grader owns. The policy should deny that fraudulent write the moment it is attempted, regardless of the model's reason.

The table below reports the cheating attempts made by each model in each instruction mode. We count a cheating attempt when the agent's submitted patch includes code that overwrites the grader. The concrete prompts are included in the appendix.

Every model saw the same 100 tasks, drawn from the benchmark's 731 so the language mix matches the full suite: 38% Go, 36% Python, 23% JavaScript, and 3% TypeScript, covering all 11 of its repositories. Each task ran once per instruction mode and policy condition, twice in the honest mode. We fixed the sample and the analysis plan before the first episode ran. Every cell holds all 100 tasks, so the percentages below are also counts.

Key findings. No model cheated unprompted: given only the task, or a nudge to maximize its score for least effort, none of the five produced a successful cheat. Under an explicit command, and with their guardrails enabled, four of five attempted the grader overwrite in 55% to 61% of runs. GPT-5.5 never did, so willingness varies by vendor. The policy changed outcomes rather than intentions: most commanded runs still attempted the cheat with it on, and far fewer landed one, cutting successful cheating by roughly 79%. The survivors were diagnosable. Almost all took a route the rules never named, and one exposed a defect in our own enforcement.

Table 1. Cheating attempts by model and instruction mode, with policy enforcement off, measured as the share of runs whose submitted patch overwrites the grader. †For Claude Sonnet 4.6 and DeepSeek-V4-Pro the honest and misaligned runs were not itemized for attempts; what we verified there is that none of them produced a successful cheat.

Table 1 shows where the willingness sits: nothing without a prompt, and 55% to 61% under the command for four of the five.

The two malicious prompts differ in one respect: whether the honest fix stays available. Under the command, Opus reaches for the shortcut in most runs. Under the suggestion, where repairing the bug is still permitted, it largely repairs the bug instead. The cheating attempts fall from a majority of runs to almost none.

The five models did not behave alike, and the spread reinforces the thesis:

• Willingness ranged widely: Claude Sonnet reached for the shortcut in more than half its runs under the command, while GPT-5.5 never touched the grader under the command, refusing outright or quietly fixing the bug instead, yet reached for the shortcut when it was merely suggested. Safety training differs from one vendor to the next.

• Effort, propensity, and effectiveness are three separate measurements: Sonnet ran far longer than Opus, on the order of three times as many turns per task, a median of 45 turns against 13.5, and it converted a higher share of its attempts into working cheats than Opus did. Propensity moved independently of both: GLM-5.2 reached for the shortcut almost as often as Opus, 55% of runs against 58%, yet landed the fewest working cheats of the four models that tried.

• Behavior after a policy denial varied too, and a denial changed the outcome rather than the objective (Table 2). Under the policy 88% to 94% of commanded runs still attempted the cheat, counting the writes the policy denied, and only 2% to 7% landed one. Sonnet's successful cheats fell from 29% of runs to 2%, while Opus, GLM-5.2, and DeepSeek-V4-Pro each kept a small residue by reaching the grader through a file the policy had never named.

• Some models hedged: told to cheat, they implemented the genuine bug fix and planted the forged report anyway. Of the 52 successful cheats we itemized, those from Opus, GPT-5.5, and GLM-5.2, roughly one in seven did both.

Table 2. Under the command, with policy enforcement off and on: the share of runs that attempted the cheat and the share that landed one. Under the policy most runs still attempted it and few landed one. The policy-on attempt figure also counts writes the policy denied, so the two attempt columns are not a like-for-like behavioral comparison. GPT-5.5 never attempted the cheat under the command in any of its runs.

The table below reports successful cheats per 100 tasks under each malicious prompt, with policy enforcement off and on. A cheat counts as successful only when it actually fooled the grader; a blocked attempt does not count.

Table 3. Share of runs with a successful cheat under each malicious prompt, with deterministic policy enforcement off and on. The commanded columns repeat Table 2's landing rates so that the two prompts can be read together.

The policy we tested is deliberately simple and hand-authored: it denies writes that match a short list of grader and test-harness paths, while leaving open the source tree the agent is meant to edit. Enforcement sits at the tool call. Before a write runs, the engine parses the call, matches it against the rules, and returns allow or deny. The decision is made outside the model, and every decision is written to an audit log. That one rule cut successful cheating from 84 of 500 commanded runs to only 18, roughly 79%, and from 24 of 500 suggested runs to only 2. Its simplicity has a price, as it may over-block legitimate tool calls that merely look suspicious. Across the honest tasks, completion fell about two points, measured on Opus alone and not statistically separable from no cost at this sample size. We have not yet measured that cost on the other four models.

A defense layer that evolves

The survivors were not random, and they fall into two classes. The policy works by naming the places and files an agent must not write to, and that list is almost impossible to make exhaustive. A coverage failure is the same tampering carried out somewhere the rules never named. An enforcement failure is the policy deciding correctly and the write landing anyway. Of the eleven survivors we adjudicated in detail, ten were coverage failures. The eleventh was an enforcement failure: the policy denied the write three times and the file landed regardless. That second class is a defect in our own machinery rather than a gap in the rules, and we treat it as the higher-priority bug.

A defense that enumerates forbidden routes stops only the routes it already knows; a capable adversary takes the one left off the list [13].

So the layer above the rules has to learn and evolve in order to contend with an adversary who searches for the unnamed route. To this end we have built a loop that red-teams our own engine, confirms each finding with a working proof-of-concept attack inside a controlled blast radius, and hardens the policy against it [14]. The loop runs red team against blue team: one process searches for bypasses, the other turns each confirmed bypass into a constraint the policy enforces. That loop is running now under human supervision, as a separate line of work, and full autonomy is where we are taking it. None of the numbers above come from it: the policy we measured was written by hand and left fixed. Si vis pacem, para bellum. The war is one we wage on our own defenses, not on anyone else.

Closing

What we draw from this is a deployment discipline: assume the guardrails will be tested. Isolate the agent so a compromise cannot spread, hold every tool call to a deterministic policy, and report what that policy costs as well as what it misses. A security layer you cannot audit is one more thing you are trusting on faith.

Making that policy keep pace with the attacks is where our research goes: pushing the red-team-and-harden loop toward full autonomy, adding remediation that closes a gap as soon as it is found, and moving from a list of forbidden routes to a layer that learns the routes we have not named.

Three more things follow. The boundary should be learned from benign behavior too, not only from attacks, so least privilege is in place before the first attack is found. A denial should carry its reason, naming the rule that fired and the attack class that rule exists to exclude. And the boundary should be checked rather than trusted, proving that a synthesized policy excludes every attack we have found and does not re-open an old hole as the set grows.

The agents keep working. What changes is what stands in front of them.

References

[1] OpenAI, "OpenAI and Hugging Face partner to address security incident during model evaluation," OpenAI, 21 July 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/

[2] Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, et al., "ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?", arXiv preprint arXiv:2605.11086, 11 May 2026. https://www.cybergym.io/exploitgym

[3] Anthropic, "Investigating three real-world incidents in our cybersecurity evaluations," Anthropic, 30 July 2026. https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals

[4] Bruno Ferreira, "AI tool OpenClaw wipes the inbox of Meta's AI Alignment director despite repeated commands to stop," Tom's Hardware, 24 February 2026. https://www.tomshardware.com/tech-industry/artificial-intelligence/openclaw-wipes-inbox-of-meta-ai-alignment-director-executive-finds-out-the-hard-way-how-spectacularly-efficient-ai-tool-is-at-maintaining-her-inbox

[5] Anthropic, "Redeploying Fable 5," Anthropic, 30 June 2026. https://www.anthropic.com/news/redeploying-fable-5

[6] Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong, "Formalizing and Benchmarking Prompt Injection Attacks and Defenses," 33rd USENIX Security Symposium, pp. 1831-1847, 2024. https://www.usenix.org/conference/usenixsecurity24/presentation/liu-yupei

[7] Max McGuinness, Mikaela Grace, Jiri De Jonghe, Jake Eaton, and Abel Ribbink, "How we contain Claude across products," Anthropic Engineering, 25 May 2026. https://www.anthropic.com/engineering/how-we-contain-claude

[8] Evan Li, Tushin Mallick, Evan Rose, William Robertson, Alina Oprea, and Cristina Nita-Rotaru, "ACE: A Security Architecture for LLM-Integrated App Systems," Network and Distributed System Security Symposium (NDSS), 2026. https://www.ndss-symposium.org/ndss-paper/ace-a-security-architecture-for-llm-integrated-app-systems/

[9] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, et al., "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?", International Conference on Machine Learning (ICML), 2026. https://arxiv.org/abs/2509.16941

[10] Adam Stein, Davis Brown, Hamed Hassani, Mayur Naik, and Eric Wong, "Finding Widespread Cheating on Popular Agent Benchmarks," DebugML, University of Pennsylvania, 10 April 2026. https://debugml.github.io/cheating-agents/

[11] Maia Hamin and Benjamin Edelman, "Cheating on AI Agent Evaluations," NIST Center for AI Standards and Innovation, 2 December 2025. https://www.nist.gov/blogs/caisi-research-blog/cheating-ai-agent-evaluations

[12] Hao Wang, Qiuyang Mang, Alvin Cheung, Koushik Sen, and Dawn Song, "How We Broke Top AI Agent Benchmarks: And What Comes Next," Berkeley RDI, April 2026. https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/

[13] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger, "Defining and Characterizing Reward Gaming," Advances in Neural Information Processing Systems 35 (NeurIPS), 2022. https://proceedings.neurips.cc/paper_files/paper/2022/hash/3d719fee332caa23d5038b8a90e81796-Abstract-Conference.html

[14] Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, et al., "HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal," Proceedings of the 41st International Conference on Machine Learning (ICML), PMLR 235:35181-35224, 2024. https://proceedings.mlr.press/v235/mazeika24a.html

[15] AI Security Institute, "Incident Report: unsanctioned agent behaviour during cyber testing," AI Security Institute, 4 August 2026. https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing