I benchmarked my cost-router and it killed itself

I was building a cost router. Classify each request, send the easy ones to the cheap model, report the savings at the end of the month. Every LLM pipeline eventually grows one. Mine was going to be measured.

One gate before writing it: is the cheap model actually cheaper per task?

It is not. Not slightly. Not in some edge case.

The run

150 paired tasks. Same prompt bytes, same flags, only the model differs. Sonnet 5 against Haiku 4.5, which lists at half Sonnet's price per token.

Two to seven times more expensive, at half the token price. Cheaper on 1 task out of 150. Quality indistinguishable — every conditional-accuracy interval overlaps its counterpart.

This is not "you get what you pay for." Both models did the work. One of them spent a fortune getting there.

Why

The obvious objection is that Haiku is verbose and a tighter prompt fixes it. The prompts were byte-identical across arms, hash-asserted in code — tightening one side's prompt to win your own comparison is exactly the fraud the project existed to avoid.

The real answer is that two orderings invert. Rank the families by how short the required answer is:

MMLU-Pro (one letter)  <  GSM8K (a number)  <  MBPP (a function)

Now rank them by how many output tokens Haiku actually spent:

GSM8K (509.9)  <  MBPP (1414.2)  <  MMLU-Pro (2255.8)

The family that needs a single letter drew the highest spend. Haiku reasons at length regardless of how short the answer has to be, and --effort low doesn't restrain it — measured, that flag moves Haiku's output by about 7% while moving Sonnet's by 9.3x.

Price per token is a number the vendor sets. Tokens per task is a number the model decides. You are billed on the product, and you only control one factor.

The part that nearly ruined it

Two bugs were caught in review, both silent, both under load-bearing claims.

A duration column used EXTRACT(MILLISECONDS FROM interval), which returns only the seconds field. A 90-second attempt recorded as 30 seconds — wrong only for the slow jobs, which are the only jobs anyone cares about. And replay() accepted a job in any state, so replaying a succeeded job minted a second job for the same event and re-ran the handler, routing around the idempotency guarantee the whole repo exists to demonstrate.

An adversarial pass before any code existed also found the original benchmark statistically incapable of passing its own test, and a pre-flight gate written backwards — it would have greenlit the expensive run precisely when routing was pointless.

I'm listing these because a benchmark post that hides its own instrumentation bugs is the thing this post is arguing against. If I'll publish the timing bug, you can trust the cost table.

What I cut

The router. Before building it.

It survives in the repo as a policy evaluated and rejected on evidence, which is a better artifact than a router that "works" because nobody measured it. The pipeline underneath got built anyway and stands on its own: idempotency enforced at the database rather than in application logic, SKIP LOCKED claims, full-jitter backoff, a lease reaper for crashed workers, a dead-letter queue that's a query rather than a second table.

Limits, because a number without them isn't a result: n=50 per family, one model pair, one substrate, --effort low only, three short-answer programmatically-graded families, no multi-turn work, pricing sensitive to scheduled changes. Dollars are derived from token counts against a checked-in price table, not billed.

Every figure recomputes offline from committed raw per-call data, no API key needed. Clone it and check.

The transferable part

The feature you're about to build has an assumption underneath it. Usually one. Mine was "cheaper per token means cheaper." It cost about a day to test and it was wrong by up to 7x.

Test the assumption before you build the thing that depends on it. Sometimes the benchmark writes the ticket that closes itself.