A better prompt beat a bigger model in my app — at a sixth of the cost

On the hard half of my test set, Claude Haiku with a good prompt matched Claude Opus with the same prompt — 86% vs 90%, error bars overlapping — and cost one-sixth as much per correct answer.

So which one should the feature ship on? That’s the question almost nobody measures, and the one your monthly bill is quietly answering for you.

The wrong default: “use the best model”

Most apps wire in the smartest model and move on. It feels safe. It’s also how you end up paying frontier prices for a job a cheaper model does just as well.

The number that actually matters isn’t cost per call. It’s cost per successful task: total dollars ÷ the number of tasks that cleared your quality bar. A model that’s cheap but wrong a third of the time isn’t cheap.

The decision rule that falls out of it:

Don’t pick the best model. Pin a quality bar, then send each task to the cheapest model (and prompt) that still clears it.

This isn’t a hunch — it’s the finding in the literature. Cost-of-Pass shows the economical model is task-difficulty-dependent, not capability-ranking-dependent: cheap models waste money on hard tasks, frontier models waste it on easy ones. FrugalGPT matched the best single model’s quality (GPT-4) with up to 98% lower cost by routing through a model cascade instead of defaulting. What’s missing is your numbers.

The rig

The most common AI-in-app feature there is: read a messy customer email, fill a structured record (name, company, product, quantity, budget, currency, date, urgency).

• A golden set, label-first. I generate the correct record first, then render the email from it — so the label is authoritative by construction, zero labeling error. 70 cases, split into a clean tier (short, explicit) and a messy tier (forwarded threads, typos, a quote and a budget where only one is the answer, dates in prose, missing fields, distractor numbers).
• A deterministic grader. No LLM-judge. Exact match for amounts, quantities, currency codes, dates and enums; a tolerant matcher for free-text names (so “SuperServer” vs “SuperServers” doesn’t count as a miss). A pre-registered bar: a case passes only if every critical field is right and ≥90% of all fields are right. Hallucinating a value where the answer is “null” is a fail — that’s the move that separates cheap models.
• Two axes, not one. Three models (Haiku, Sonnet, Opus) × four prompt scaffolds: S0 naive, S1 + a JSON schema, S2 + per-field rules, S3 + a couple of worked examples. Twelve configurations, every case run multiple times.
• Comparable cost. Caching off and asserted off on every call, so the per-call number is order-independent. Cost computed from the API’s own token counts × a dated price list — not a tokenizer estimate, not the bill. Same grader, same knobs, for everyone.

What the numbers said

Clean tier: everyone scored 100%. So the cheapest setup won outright — Haiku, no schema, 100% pass at $0.0008 a task. On easy work, paying for intelligence is pure waste.

Messy tier: where it pays off. Two results:

• A better prompt beat a bigger model on quality and price: Haiku + a schema cleared 83% at $0.0026; Opus naive cleared 80% at $0.0056.
• A cheap model nearly caught the frontier once both had the good prompt: Haiku + examples 86% vs Opus 90% (overlapping error bars) — at $0.0030 vs $0.0168, 5.6× cheaper.

(More scaffolding wasn’t always better — a couple of rungs backfired — but that’s the technical companion’s rabbit hole. And the schema-tier dollars carry some harness overhead, so compare the shape, not the exact cents.)

The playbook you can copy

You don’t need my rig. You need the loop:

1. Take ~20 real tasks from the actual feature — including the ugly ones that break things.
2. Write down what “good” means as a check a script can run.
3. Sweep the grid — a few models × a few prompts — and read cost per successful task.
4. Route by difficulty. Send the easy bulk to the cheapest model that clears the bar; keep the expensive one for the cases that genuinely need it.

It’s the same discipline that makes any AI build defensible: measure on your own task, not the launch-day chart.

So what

A weekend eval like this turns “use the best model, to be safe” into a number you can defend, and usually into a smaller invoice for the same result. The cost isn’t the model you chose; it’s the comparison you skipped.

If someone handed you a cheaper model and a better prompt tomorrow, could you prove — with your own tasks — that your output got worse? Or just that your invoice did?