The harness spends your money, not the model

Same model. Same spec. One coding agent burned roughly five times the tokens of another to build the identical app — and a third never reached a working build at all.

The model wasn’t the variable. The harness was.

We argue about models like they’re the whole story — Sonnet vs Opus vs GPT. But the thing you actually choose, the thing that quietly decides your bill and half your output quality, is the harness: the agent wrapped around the weights. The loop. The file-edit format. How it manages context, when it retries, whether it plans. METR named the layer years ago — “we call scaffolding + model combinations agents” — and the data since keeps saying the scaffolding does more of the work than we admit.

The model is the commodity. The harness is the variable.

The numbers are not subtle.

• Same model, scaffold swap, +10 points. Epoch AI ran GPT-4o on SWE-bench Verified under two scaffolds: 23% with one, 33.2% with another — no model change, just the harness. Their verdict: “performance reflects the sophistication of the scaffold as much as the capability of the underlying model.” (Epoch AI)
• Six harnesses, one model pool, a 24-point spread. Harness-Bench held the models fixed and varied only the harness: best 76.2%, worst 52.4% — and token use ranged from 93.7K to 175.1K for the same work, with the top scorer spending fewer tokens. More trajectory ≠ better.
• The harness out-weighs the model. A 2026 study measuring both found harness choice explained roughly 7.8x the variance that swapping the model did. (arXiv)
• Even the edit format is a lever. In Aider’s benchmark, putting a model in architect/editor mode lifted o1-mini from 61.1% to 71.4% — same model, different loop. And “whole-file” editing burns far more tokens than diff editing for similar accuracy. (Aider)

Anthropic makes the same point from the other side: Opus at medium effort matches the previous model’s score using 76% fewer output tokens.

The twist: with tests, everyone passes — so cost becomes the whole story

If you hand every agent a fixed spec and a fixed test suite — the way you would with TDD — then “did it pass?” mostly stops being interesting. Given enough turns, most decent harnesses drag the tests to green. Same destination.

What doesn’t converge is what it cost to get there, and what the code looks like underneath. One agent reaches green in 300 clean lines; another in 1,800 lines of slop that passes the visible tests and quietly fails the ones you held back.

So the question isn’t “which agent is smartest.” It’s “which agent gets my team to the same result for a quarter of the cost and code I’d still want next month.”

So I’m running it — properly

Claims are cheap. I built a rig to measure it.

• One frozen spec. A real, shippable link shortener — create links, custom aliases, redirects, click counts, expiry. Useful enough to deploy, small enough to build dozens of times.
• One black-box oracle. ~30 HTTP tests that don’t care what framework or file layout an agent picks — they only check behaviour. The agent makes them green; it may not edit them. Plus a held-out set it never sees, to catch teaching-to-the-test.
• One uniform meter. Every harness routes through a local proxy so tokens (input / output / cache) are counted the same way — not trusting each tool’s own number. Cost is computed from one price list, so the dollars are comparable.
• The variable is the harness, and only the harness. Same pinned model everywhere. Three runs each — a single run can flip the ranking, so I report median and range, not a hero number.

I measure: wall-clock, dollars, tokens (and the cache-read ratio — a big driver of the gap), turns, tests passed, held-out tests passed, lines of code, dependency count, and whether it survives a type-check. The hypothesis: same model, same spec, roughly the same result — wildly different cost. I expect at least one agent to disprove the “same result” half by shipping something that passes the demo and rots.

The harnesses worth testing right now

By adoption (JetBrains’ 2026 developer survey, plus install counts):

• Claude Code — the fastest-growing of the lot, ~18% of developers using it at work, up 6x in a year. My daily driver; I’ll run it both vanilla and with a TDD skill layer, to isolate what the skill itself buys. (JetBrains)
• Cursor — the commercial heavyweight ($1.2B ARR), now a full agent platform.
• Cline — the most-installed open-source agent (~5M installs). Full-file rewrites make it a perfect token contrast against diff-based tools.
• Aider — the diff-editing original. Cleanest to measure and pin, and the efficiency baseline everything else is judged against.

GitHub Copilot is bigger than any of them by raw users, but it’s still mostly an in-editor assistant rather than an autonomous agent you hand a spec — a different category for this test.

So what

Your AI tool choice is a budget decision wearing a capability costume. The real edge was never the model and the debt was never the AI’s fault — both come down to the loop you build around the thing.

Results next: same spec, four agents, one bill each.

If you handed your current agent and a cheaper one the same spec tomorrow — would you bet the result is identical, or just the invoice?