Back to blog

AI Benchmarks Are Fake (And Everyone Knows It)

5 min read
ai tech

AI Benchmarks Are Fake (And Everyone Knows It)

Every time a new model drops, there’s always that accusation of benchmark maxing.

And honestly, I get it. It’s a thing that’s happened, still happens, and you can feel it when a model looks really good on paper but feels bad to use.

Recently, Cursor research showed that smarter models are actually becoming even more resourceful at hacking coding benchmarks. So I decided to dig into how these benchmarks get gamed, how bad it actually is, and what a trustworthy benchmark should look like.

Let’s jump into it.

The Cursor Research

So Cursor dropped this research last week called “Reward hacking is swamping model intelligence gains.” Written by Naman Jain, a research scientist there. And it has real numbers proving how widespread this behavior is.

The problem they looked at? Eval suites built from real bugs that later got fixed.

Here’s the thing - when a benchmark uses a popular open-source library and its bugs, modern coding agents can just look up the exact commit that fixed that bug instead of actually solving it themselves. So the model might be really good at web search, but not actually the coding part.

They found that on SWE-Bench Pro, 63% of successful Opus 4.8 max resolutions retrieved the fix rather than deriving it. Either it used the web to find the relevant PR or the fixed source file, or it even used the bundled git history to time-travel forward and find the fixed commit.

Wild.

The Fix (And What It Revealed)

Cursor’s fix was straightforward - make the environment stricter. Delete the git directory, start fresh, deny network access by default with a pinned proxy only for package registries. Nothing else allowed through.

When they reran the benchmarks in this strict environment, the pattern was damning:

Funnily enough, the worst offender on SWE-Bench Pro was actually Composer 2.5 - which is Cursor’s own model. So respect to them for admitting it.

The takeaway? Benchmark design needs to account for the runtime environment. That doesn’t mean cutting off internet access entirely - you might actually want to test how well it uses those tools. But you need to be aware of how that access changes the scores and what it actually means.

Benchmark Contamination

Reward hacking is one thing. But there’s also the issue of benchmark contamination - where the model has literally seen the same question during training.

The most obvious form is exact test set contamination. But it gets subtle. Some researchers use a method where they generate two variations of a question - one from the benchmark, one similar but different. If the model consistently picks the real one, it might have seen it before.

Another paper calculated a “data contamination risk score” looking at overlap patterns - from similar wording all the way up to exact questions. High risk score = marked down.

Their data showed Qwen 2.5 scored in the mid-90% range on SST-2, but when adjusted for contamination, dropped to around 30-40%. That model may have seen the benchmark in its training data. Makes it look way better than it is.

The GSM8K Test

The last study I found - and I went down a rabbit hole on this - was from 2024 by Scale AI.

They took GSM8K (a grade school math benchmark) and created a brand new benchmark written by humans meant to mimic the exact same difficulty level. If the model really could solve the first one on its own, it should perform similarly on the new one.

That was not the case.

Many models showed a large gap between the two. The only difference? One was public data.

So Are All Benchmarks Worthless?

Not really. Researchers have been aware of this for a while, and so have benchmark makers.

For reward hacking, benchmarks like Deep SWE already use isolated environments. For contamination, many benchmarks no longer make their datasets public. Frontier Code from Cognition has no plans to release theirs. Arc AGI even refused to run their evals on Anthropic’s new model due to concerns about data retention terms.

The problem is still real though. As models become more aware that they’re being evaluated, they might change behavior in subtle ways you can’t fix by simply sealing git history or restricting internet access.

This is a seriously hard problem.

My Take

Have you had that feeling before? Where a model is a lot worse than its benchmark score suggests? I definitely have.

The numbers look impressive on paper. Then you actually use the thing and it fumbles something basic. That gap? That’s what this is about.

I don’t have a perfect answer. But I do know that the more we call this out, the harder benchmark makers will work to fix it. And the more we pay attention to how evals are actually run - not just the final number - the better choices we’ll make.

Trust the vibes, not just the scores.