Intelligence is rising, the leaderboard is lying, and the harness still decides

Three things are true at the same time right now, and most takes pick one and drop the other two.

1. AI models are getting more capable, fast — on a curve clean enough to fit a line to.
2. The leaderboards everyone quotes are increasingly gamed and contaminated.
3. The harness around a model — the agent loop, the tools, the prompts — can move its score by more than a whole model generation.

Hold all three at once and you get a more useful question than “which model is smartest?“

1. The capability curve is real, and it’s steepening

The cleanest measure of “is AI actually getting better” isn’t a quiz score. It’s METR’s task time horizon: the length of task — measured by how long a human expert takes — that a model can finish on its own, half the time.

That number has doubled roughly every seven months since 2019, and on models from 2023 onward the doubling time has tightened to about four months. The anchors, from METR’s own table: GPT-4 sat at 3.5 minutes of autonomous work. Claude 3.7 Sonnet, 60 minutes. o3, two hours. GPT-5, three and a half. Claude Opus 4.5, over five hours. GPT-2, for reference, managed about two seconds.

This isn’t one lab’s artifact. Epoch AI tracks a broad capabilities index across 149 models and finds a statistical breakpoint in April 2024: frontier progress jumped from 8.3 to 15.5 points a year — it nearly doubled its own rate. Two different instruments, same shape.

So the exponential is the part the skeptics get wrong. Now the part the boosters get wrong.

2. The leaderboard is lying — at least the public ones

The benchmarks in the launch-day chart are not measuring what you think.

Start with contamination. Scale AI built GSM1k — a fresh set of grade-school math problems matched item-for-item to the famous GSM8k — and re-ran the models. Accuracy dropped by up to 8%, with several model families showing “systematic overfitting.” The worst offenders were models that had, in effect, seen the test; frontier models that hadn’t barely moved. When the test set leaks into training data, the score measures memory, not reasoning.

Then there’s outright gaming. “The Leaderboard Illusion”, a 2025 study, found Meta privately tested 27 variants of Llama-4 on Chatbot Arena before release, disclosing only the best. The punchline came at launch: Meta floated a chat-tuned “experimental” Maverick that shot up the rankings; the model it actually shipped landed 32nd, below GPT-4o and Claude 3.5 Sonnet.

And much of the canon is simply saturated. GPT-4 hit 86% on MMLU back in early 2023, and every frontier model since lands within a few points of it — close enough that the gap between this month’s model and last quarter’s, on the benchmarks in the slide deck, is mostly noise.

3. The harness moves the score as much as the model

You can hold a model’s weights fixed and change only the scaffold around it — the agent loop, the tools it can call, how the task is framed — and, as far as the benchmark is concerned, end up with a different model.

Princeton’s Holistic Agent Leaderboard ran exactly that test: 21,730 agent runs, same models, different harnesses. On SWE-bench Verified Mini (a subset), the same Claude Sonnet 4.5 scored 68% under one scaffold and 34% under another. GPT-5, on medium reasoning, went from 46% to 12%. A 34-point swing with no change to the model — and that gap dwarfs the 2–4 points that usually separate one model release from the next.

The better scaffold isn’t free, either: Sonnet’s 68% run cost about four times its 34% one. The harness sets your bill as well as your score.

The upside is just as large — a good harness unlocks a model. Princeton’s SWE-agent gave GPT-4 a purpose-built interface to the codebase and reached 12.5% on SWE-bench, far past what the same class of model managed with plain prompting; the Agentless pipeline, with no agent loop at all, scored 32% on SWE-bench Lite at $0.70 a fix. Anthropic says the quiet part in writing: “the performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.” Their fix was to keep the scaffold minimal — two tools, Bash and Edit — and let the model drive. It got Claude 3.5 Sonnet to 49%.

So what — if you’re deciding what to build on

Put the three together and the leaderboard ranking is close to the least useful number in the room: capability is rising for everyone, the public scores are gamed, and the lever that actually decides what you ship — the harness — was never on the chart.

That’s not bad news. It’s the opposite: the part that decides your result is the part you build, not the model you rent. The model is a layer you swap in an afternoon. The harness — the tools you wire up, the eval you run on your task, the loop you tune — is the moat.

So the question to answer before the next model launch tempts you: have you ever scored your own workload, on your own harness, instead of trusting the launch-day chart? Because that number is the only one that predicts what you’ll get.