Your AI Agent's Code Passes Every Test and Is Still Insecure 3 Times Out of 4

Green tests no longer tell you the code is safe. A paper posted to arXiv on October 10, Last Secure Code Benchmark: Is the Code You Vibe Secure?, measured the gap, and it is wide.

The authors built 450 tasks from 150 real vulnerabilities across seven languages (Java, C, C++, JavaScript, Python, PHP, and Go) and 24 weakness types, drawn largely from the 2025 CWE Top 25. Six frontier coding agents got the kind of prompt you write every day: build this feature. The prompts never mentioned security.

Hidden functional tests and hidden security tests then graded each result. Code had to pass both to count.

The numbers

Functional pass rates across the six agents ran from 73.6% to 91.3%. Code that was functional and secure topped out at 25.1%, from Claude Fable 5. The other five landed between 16.4% and 24.7%.

So the best agent in the study ships a working but exploitable result roughly three times out of four when nobody asks for security. The tests you already run will not catch that, because they check what the code does and not what an attacker can make it do.

One caveat: the paper tested Fable 5, and Anthropic has since released Fable 5.1. Newer models may score differently. Do not assume they score much better until someone reruns the benchmark.

What to do about it

Ranked by how much each one buys you:

1. Put security requirements in the spec. The benchmark failed because the prompt said nothing about security. State input validation, authorization checks, and forbidden patterns as acceptance criteria, in the same document the agent reads before it writes code.
2. Add exploit-style tests, not just happy-path tests. The benchmark replays the original attack against the generated code. Do the same for your own past incidents and for the weakness classes in your stack.
3. Gate merges on a security check the agent cannot edit. If the agent can modify the test or the scanner config, the gate is decoration.
4. Review agent diffs for the boring classes first. Injection, path traversal, and missing authorization checks are where real-world exploits cluster, and they are what this benchmark replays.
5. Stop reading pass rates as safety rates. A vendor claiming 90% on a coding benchmark is telling you about functionality only.

The spec is the cheapest control you have

Every step above gets cheaper when requirements are written down before the agent starts. A spec-driven workflow makes "must reject unauthenticated writes" a line item the agent is held to, instead of a hope in a reviewer's head.

If your agent's prompt does not mention security, assume the output is insecure and plan the review accordingly.