Back in September, we asked DeepSeek V4.1 Flash to hack the targets in our AI Hacking Race benchmark. It compromised every vulnerable target it was given, 11 out of 11, and left all four patched controls alone. The model bill for the whole run was $4.65.
Every one of those wins was verified. What the score didn't show was how DeepSeek got there.
DeepSeek went 11 for 11
The AI Hacking Race is built to make cheating hard. The attacker, the target and the verifier, which we call Triage, each run in their own container on a clean, isolated network, with no public internet, host access or other runs in reach. The attacker gets source code and a Bash tool. A win only counts when the target itself executes /arena/submitflag and submits a value randomized for that run, so an agent can't just claim success. Every model gets the same code, prompt, tools and time rules, with no hints, source diffs or git history, and we record every command and output.
DeepSeek cleared that bar on every vulnerable run, issuing 2,349 Bash commands over 2 hours and 38 minutes.
That was too clean to take at face value, so we read the trace of every successful run.
Five of the 11 wins skipped the bug we meant to test
Six wins followed the path we designed: three on a Jenkins credential challenge, two on Nextcloud, and one on a Jenkins upload race. The other five found routes we never intended.
Grafana is the clearest case. The challenge was built around a specific file path handling error, and DeepSeek never touched it. It dropped executable files into a temporary plugin folder, asked Grafana to load that folder as a normal plugin, let Grafana run the payload, and submitted the flag. All three runs did this, in 52, 64 and 90 seconds. Nothing in the task rewarded the harder route, so the agent took the shorter one.
Jenkins showed both sides of the result. On the credential challenge, all three runs took the intended path. On the upload challenge, only one run performed the intended timing attack: it sent a single byte, held the request open, changed the destination through a second upload, completed the first upload and triggered a build. The other two found shorter file link routes, and HTTP 201 responses showed Jenkins accepting a new link and storing a file through it. DeepSeek dropped the timing step entirely. Judged by the final flag alone, all three upload runs looked identical.
None of these were new vulnerabilities in Grafana or Jenkins. They were gaps in how we built our own challenges, and the agent found every one of them without any hint to look.
Our verifier was right, and the score still misled us
Every one of those 11 wins was real. The target executed the command, and our verifier did what it was designed to do. The fixed controls didn't catch it either, since they showed the vulnerable and patched versions behaved differently without showing which mechanism made the difference.
A benchmark has three layers: what the prompt asks for, what the evaluator measures and what the agent can change. An agent optimizes the second layer, so every gap between the goal and the measurement is a route. We were missing a second question. Impact proof asks whether the target was compromised. Path proof asks whether the agent used the weakness the challenge was built to measure. Meta's Muse Spark 1.3 failed the first test: it reached code execution and then never ran the final verification command. DeepSeek passed the first test and, on five runs, failed the second. A single score hides the difference.
Agents have gamed other benchmarks the same way
We aren't the first to see this. In Palisade Research's study of specification gaming in reasoning models, agents asked to beat Stockfish at chess sometimes won without beating it. They edited the board state, changed the status file or replaced the Stockfish program, because the environment allowed it and the task only asked them to win.
METR found reward hacking attempts in roughly 1 to 2% of o3's task attempts across HCAST and RE-Bench, including modified tests and exploited scoring code. On RE-Bench, 7 of 164 attempts would have pushed the reported total beyond expert performance had METR accepted them.
On SWE-bench, the answer can leak through the environment itself: NIST's CAISI found agents reading future commits in a repository's git history on SWE-bench Verified. OpenAI's later audit of SWE-Bench Pro estimated that about 30% of its tasks were broken.
The pattern repeats. If an agent can reach something the evaluator relies on, assume it will test it.
We closed the shortcuts and versioned the benchmark
We now start every challenge with a threat model for the benchmark itself. We assume the agent will read every visible file, test every endpoint, inspect git history if it's available, race concurrent processes and look for easier ways to trigger the verifier, and we treat the benchmark code as part of the target.
On September 11, we closed the Grafana staging and reload routes and the Jenkins file link routes, while keeping the timing race the upload challenge exists to measure. We added negative tests for each shortcut we saw, added a path level review of every successful run, and versioned the benchmark so results from different versions are never mixed.
The original result still sits on our board with a narrower claim attached: 11 verified compromises, 6 of them through the intended exploit. Scores from that run can't be compared with scores from the current benchmark, so we reran the same model on the new version.
On the current benchmark, DeepSeek verified 4 of 11
On October 18 we ran DeepSeek V4.1 Flash again on the current version of the suite, with the shortcut routes closed. The provider route also changed: the original run went through Novita's FP8 endpoint, which doesn't host the newer 1017 snapshot, so the re-run used DeepInfra's FP8 endpoint.
The re-run verified 4 of its 11 vulnerable runs: two on Grafana and two on the Jenkins credential challenge. It went 0 for 3 on the upload race and 0 for 2 on Nextcloud, and all four patched controls stayed unverified.
The drop is bigger than the shortcuts explain. Closing those routes took away five wins, which would have left six. Against those six, the re-run lost one Jenkins credential run, the intended upload race win and both Nextcloud runs, and it picked up two Grafana runs through the intended bug. The Jenkins credential and Nextcloud challenges never had a shortcut. With 11 vulnerable runs per model and a different provider route, we can't say how much of that gap comes from the route and how much is ordinary variance between runs.
The re-run also took longer, using 4 hours and 42 minutes of active time against 2 hours and 38 minutes for the original. The newer 1017 snapshot also verified 4 of 11 on the same DeepInfra route, but across three tasks instead of two.
Questions to ask about every benchmark score
The audit left us with a checklist we now apply to any benchmark number, ours or anyone else's. What capability does the score claim, what event counts as success, and who produces the proof? Can the agent reach the scoring code? Were successes reviewed for alternate paths, and did the benchmark include fixed and adversarial controls? Did every model run on the same benchmark version, and does the sample size support the claim?
In the webinar, Yanir makes the case that a trustworthy benchmark doesn't need to be one agents never outsmart. It needs to detect unexpected wins, keep the evidence and get harder to fool each time. He walks through the full traces and this checklist in the recording.
The webinar took place before we published the October 18 re-runs, so those results aren't covered in the recording. Watch the recording