Everybody wants to scare you that the next AI model will hack the planet. As if cybersecurity wasn’t already full of fearmongering by people who’ve never touched a terminal in their life.
Exploiting a bug is not the same as finding one
Most cyber benchmarks test two things:
Exploitation: models receive information about an existing vulnerability and are tasked with creating an exploit. Some examples CyberGym level 1, ExploitGym, ExploitBench.
Vulnerability discovery: can models identify vulns, and then exploit them? Examples include CyberGym level 0, CVE-Bench zero-day mode.
If we look at the model system cards from OpenAI and Anthropic, the benchmarks they are pushing are mostly exploitation benchmarks, not vulnerability discovery. If any vulnerability discovery benchmarks are mentioned, they are usually internal and not public.
The obvious question is: which one actually matters in the real world? Discovery.
If you care about getting actually hacked, it’s obviously vulnerability discovery. The hard part of hacking an enterprise has never been exploiting a bug once someone hands you the location and instructions. The hard part is finding the bug in the first place.
A known vulnerability will get exploited with or without AI
I’d argue that exploitation capability does not even unlock any substantial difference in real-world hacking. If an exploitable vulnerability is known, and someone is determined to exploit it, it will happen, with or without a capable model.
The benchmarks are focused on exploiting memory safety type bugs, and in the meantime there are vulnerabilities so simple, everywhere.
I found a vulnerability because I got annoyed
I’ve once identified a critical bug that exposed customer data, but it worked only in one out of three requests. I discovered it only because I was frustrated enough to hit the send button three times. How do you teach a model to do that? How do you benchmark curiosity, frustration, or the instinct that something feels off?
Models can copy the shot. They can’t see what led to it
These benchmarks are like judging whether someone can shoot a basketball by showing them where the ball went through the hoop. Like watching the best NBA players shoot and then mimicking them, but basketball is more than just shooting. You only see the end result, the shot that hits the net, and you have no access to their thoughts and years of experience. What made the player step to the left and not to the right?
The benchmark captures the result. It misses the judgment that produced it.
That’s exactly the distinction between a model trying to hack an enterprise versus a talented hacker. The model knows what successful attempts looked like before. The researcher knows where to look when nobody has provided the answer.
Understanding code is not the same as finding vulnerabilities
This applies to almost any field. We just happen to see cybersecurity capabilities being pushed first because models are very good at reasoning about code, which is essential to writing better code.
But developers reason about code every day. That does not make every developer a security researcher.
I was a software developer too before I transitioned to vulnerability research. It still took me months of effort to find my first real vulnerability. Finding bugs requires a different way of thinking. You question assumptions or notice strange behavior. You go down paths that look pointless until they aren’t. That skill is not the same as understanding what the code is supposed to do. It’s a whole different domain than understanding code.
Can you teach something you can’t fully explain?
Think of something you’re exceptional at. Now try to explain step by step, decision by decision, how you complete that task end-to-end. Can you? It’s hard, almost impossible.
That’s the challenge with tacit knowledge, the knowledge you have but can’t really put into words. And words are what drives LLMs. Your mind knows more than it can synthesize into language.
That’s one beauty of the human mind. And until the point you can translate Michael Jordan’s basketball skill into words, benchmarks will remain an imitation.