I ran a 2,520-call model eval on my Claude subscription — no API key, real per-call cost

I wanted one number for a real feature — which model and prompt should this ship on? — and I wanted it without guessing and without a surprise API bill. I got it: 2,520 graded calls, $0 of metered spend, a cost figure for every one.

The feature is the most common AI-in-app job there is: read a messy sales email, return a structured record (name, company, product, quantity, budget, currency, date, urgency). The question is right-sizing — the cheapest model-and-prompt that still clears a quality bar. The plain-English version of the result is a separate post; this one is the methodology.

1. A golden set you can trust: label-first

The usual eval failure is a fuzzy oracle — you label real data by hand and inherit your own mistakes. I inverted it. Generate the correct record first, then render the email from it, deterministically and seeded:

label  = { customer_name: "Hassan", company: "Massive Dynamic", product: "SuperServer",
           quantity: 21, budget_amount: 13900, currency: "GBP", ... }
email  = render(label)  →  "hey need 21 of the SuperServer. you quoted 15300 gbp but our
                            budget is only 13900 gbp. no rush. - Hassan, Massive Dynamic"

The label is authoritative by construction — zero labeling error, no eval-reality gap. 70 cases, two tiers:

• clean — short, every field explicit.
• messy — forwarded threads, typos, a quote and a budget where only one is the answer, dates in prose, missing fields (the label is null), distractor numbers (PO, invoice, phone). Perturbations corrupt the surface text, never the label’s truth.

2. A deterministic grader — no LLM-judge

An LLM-judge is expensive and drifts. A code grader is free and reproducible, so I wrote one with two regimes:

• Exact for the things that have one right form: quantity, budget, currency code, enum, ISO date (parsed tolerantly, then compared exactly).
• Tolerant for free-text names: lowercase, fold punctuation/whitespace, singular↔plural, token-set equality. Without it, "SuperServer" vs "SuperServers" reads as a miss and a critical field flips.
• Null discipline: emitting a value where the answer is null is a fail. That single rule is the sharpest discriminator between cheap and frontier models on messy input.

The pre-registered bar: a case passes only if every critical field is right and ≥90% of all nine fields are right. With nine fields that rounds to all-nine — strict on purpose.

3. The trick: run it on the subscription, not a metered key

Halfway through, my metered API key ran dry. Instead of topping it up, I moved every call to the Claude Code subscription via headless mode (claude -p). The catch: claude -p is an agent — it ships a ~23,000-token system-and-tools prefix on every call, which would swamp a ~300-token extraction and make per-model cost meaningless.

So I stripped it back to a bare model call:

claude -p "$USER_PROMPT" \
  --system-prompt "$SYS" \        # REPLACE the agent prompt (not --append)
  --tools "" \                    # no tools
  --strict-mcp-config \           # ignore all MCP servers
  --model claude-haiku-4-5 \      # the model under test
  --max-turns 1 \
  --output-format stream-json --verbose \
  --json-schema "$SCHEMA"         # S1+ only — structured output
# run from an EMPTY dir so no CLAUDE.md is auto-discovered
# env: DISABLE_PROMPT_CACHING=1   MAX_THINKING_TOKENS=0

That drops the prefix from 23,414 → ~320 tokens — a clean app-style call, billed to the subscription. Three flags do the load-bearing work:

• --system-prompt + --tools "" + --strict-mcp-config + empty cwd — remove the agent identity, the tool schemas, the MCP config, and the project CLAUDE.md. What’s left is your prompt.
• DISABLE_PROMPT_CACHING=1 — Claude Code auto-caches aggressively (I caught Opus writing a 1,071-token cache entry), and cross-run cache reuse makes per-call cost order-dependent. I assert cache_creation == 0 && cache_read == 0 on every call to catch leaks.
• MAX_THINKING_TOKENS=0 — extraction isn’t a reasoning task; Haiku otherwise burns output tokens thinking.

Structured output in one turn. --json-schema makes Claude Code call an internal StructuredOutput tool. Let it run its natural course and it loops for three turns (tool call → result → a chatty prose summary), inflating input to ~4,000 tokens and wrapping the JSON in markdown. Instead I cap it at --max-turns 1 and read the structured object straight out of the tool_use block’s .input from the stream — one model call, ~1,100 tokens, clean JSON. The cap makes the CLI exit non-zero (error_max_turns) after it has emitted the tool call, so I read stdout off the failed exit. Slightly grubby; entirely reliable.

The billing asterisk (June 2026). This is free only because claude -p draws from your Claude subscription rather than a metered key — and Anthropic nearly changed that. On May 14, 2026 it announced Agent SDK and claude -p usage would leave subscription pools on June 15 for a separate monthly credit billed at full API rates ($20 Pro / $100 Max 5× / $200 Max 20×), framed as removing a “15–30× subsidy.” After a loud backlash it paused the change on June 15 — headless usage “continues drawing from your subscription limits exactly as before,” with a reworked plan promised on notice. So as of writing this runs free on your plan; it’s also visibly in flux, so check before you lean on it for a big batch.

4. Computing the cost

Cost is computed from the response’s own usage object × a dated price list (Haiku $1/$5, Sonnet $3/$15, Opus $5/$25 per 1M in/out) — never a tokenizer estimate, never the bill. Caching off, thinking off, fixed everything else, same grader for all twelve configs. The subscription’s notional total_cost_usd is ignored; the token counts are identical whether you pay metered or not.

Add up the whole study — calibration plus the full run, 3,240 graded calls, 3.4M input + 0.67M output tokens — and the same traffic at metered API rates would have run about $21. On the subscription it was $0 extra.

One limit worth flagging: the --json-schema path carries some Claude Code overhead, so the absolute S1–S3 dollars run a bit higher than the same call on the raw API would. The shape — S0 cheap, schema tiers heavier, the ranking between configs — holds; treat the schema-tier cents as a ceiling.

5. The numbers

Three models × four scaffolds (S0 naive · S1 + JSON schema · S2 + per-field rules · S3 + worked examples) × 70 cases × k=3. Bar 0.9.

model	scaffold	messy pass (95% CI)	messy CPST	clean CPST	avg in→out tok	turns
Haiku	S0	59% (44–74)	$0.0014	$0.0008	311→106	1
Haiku	S1	83% (70–93)	$0.0026	$0.0021	1064→215	2
Haiku	S2	78% (65–90)	$0.0030	$0.0023	1258→216	2
Haiku	S3	86% (73–95)	$0.0030	$0.0026	1515→215	2
Sonnet	S0	56% (41–71)	$0.0039	$0.0019	312→75	1
Sonnet	S1	83% (70–93)	$0.0085	$0.0065	1065→238	2
Sonnet	S2	70% (55–84)	$0.0112	$0.0072	1259→247	2
Sonnet	S3	85% (71–94)	$0.0101	$0.0079	1516→247	2
Opus	S0	80% (68–90)	$0.0056	$0.0043	412→94	1
Opus	S1	68% (53–82)	$0.0181	$0.0118	1047→271	2
Opus	S2	65% (50–79)	$0.0209	$0.0131	1306→272	2
Opus	S3	90% (80–99)	$0.0168	$0.0149	1665→268	2

CPST = total $ ÷ trials clearing the bar. CIs are 95% bootstrap, clustered by case. Read it three ways:

1. Easy work: don’t pay for intelligence. Every config is 100% on clean, so Haiku S0 at $0.0008 wins and everything above it is waste.
2. A better prompt beats a bigger model. Haiku S1 (83% messy, $0.0026) outscores and undercuts Opus S0 (80%, $0.0056).
3. Scaffolding isn’t monotonic. Per-field rules (S2) drop below a plain schema (S1) for all three models, and Opus actively regresses with a schema (S1 68% < S0 80% on messy) until examples (S3) pull it back to 90%. Worked examples beat written rules. Some of this is the structured-output path; the cross-model S2 dip is too consistent to be only that.

6. Three bugs that nearly faked the result

• The grader artifact. Exact-matching product flagged "SuperServer" vs "SuperServers" as wrong, failed a critical field, and inverted the model ranking. Fixed with the tolerant matcher (§2). Lesson: a too-strict oracle is as wrong as a too-loose one.
• A data bug wearing a difficulty costume. My first calibration said the strict 0.9 bar was useless — every config collapsed to the same messy score. Per-field analysis showed one field, email, failing 36/36 times across every model including Opus. The generator was injecting a forwarder’s address into the From: header while the label said null — an unwinnable case, not a hard one. Fixed the generator; the bar separated cleanly. Lesson: when everyone fails a cell identically, suspect the data, not the models.
• Cache contamination. The subscription path silently cached prefixes, making cost order-dependent — invisible until I asserted the usage cache fields to zero and watched them not be. DISABLE_PROMPT_CACHING=1 (§3) fixed it.

Tips, distilled

• Eval on your Claude subscription with stripped claude -p — no metered key, and the usage object gives you billing-grade token counts to price yourself.
• Label-first data: generate the answer, render the input from it.
• Deterministic grader: exact for codes/amounts/dates, tolerant for free text, fail on hallucinated nulls.
• Pre-register the bar and calibrate it on a tiny grid first — it should land where models actually differ, not where everything passes or everything fails.
• Assert cache off every call; thinking off for non-reasoning tasks.
• Capture structured output at turn 1, not after the agent’s summary loop.

So what, for the technical reader

A defensible model decision is a weekend of plumbing, not a vendor chart — and you can run the whole thing on a subscription you already pay for. The rig is reusable: swap the schema and the golden set and you have an eval for your feature. The model is a dropdown; the eval is the asset.

What would your own feature’s table look like — and would the cheapest passing row be the one you’re shipping on today?