Agentic Pipeline

Repo: github.com/nikolas-sapa/agentic-pipeline

What it is

A Postgres-backed job pipeline for LLM requests — idempotent ingress, leased workers, retry taxonomy, dead-letter queue, replay — plus the benchmark that cut the feature it was built around.

The plan was a cost router: classify each request, send the easy ones to a cheaper model, report the savings. Before building it, one gate had to pass. Is the cheap model actually cheaper per task?

It is not.

The finding

150 paired tasks. Same prompt bytes, same flags, only the model differs.

Haiku 4.5 lists at half Sonnet 5's price per token. It still cost two to seven times more per task, and came out cheaper on 1 of 150 tasks. Quality was indistinguishable — every conditional-accuracy interval overlaps its counterpart. This isn't "you get what you pay for." It's a pricing-intuition failure: cheap per token is not cheap per task, because the cheap model spends far more tokens reaching the same answer.

The mechanism

The obvious objection is that Haiku is just verbose, and a tighter output contract would fix it. The contract was already byte-identical across arms — hash-asserted in code, because tightening one arm's prompt to win the comparison is the exact fraud the project existed to avoid.

The stronger answer is that the two orderings invert. Ranked by how short the required answer is:

MMLU-Pro (one letter)  <  GSM8K (a number)  <  MBPP (a function)

Ranked by how many output tokens Haiku actually spent:

GSM8K (509.9)  <  MBPP (1414.2)  <  MMLU-Pro (2255.8)

The family demanding the shortest possible answer drew the highest spend. Haiku reasons at length regardless of how short the answer must be, and --effort low doesn't restrain it — measured, that flag moves Haiku's output by about 7% while moving Sonnet's by 9.3x.

What that changed

The router was cut. It survives in the repo as a policy evaluated and rejected on evidence, which is a better artifact than a router that "works" because nobody measured it.

The pipeline was built anyway, and stands on its own: idempotency enforced at the database rather than in application logic, SKIP LOCKED claims, full-jitter backoff, a lease reaper for crashed workers, and a dead-letter queue that is a query rather than a second table.

On being wrong carefully

Two bugs were caught in review, both silent, both in load-bearing claims. A duration column used EXTRACT(MILLISECONDS FROM interval), which returns only the seconds field — a 90-second attempt recorded as 30 seconds, wrong only for the slow jobs anyone cares about. And replay() accepted a job in any state, so replaying a succeeded job minted a second job for the same event and re-ran the handler, routing around the idempotency guarantee the whole repo exists to demonstrate.

An adversarial review pass before any code existed also found the original benchmark statistically incapable of passing its own test, and a pre-flight gate written backwards — it would have greenlit the expensive run precisely when routing was pointless.

Limits

Stated on the repo as well, because a number without its limits isn't a result: n=50 per family, one model pair, one substrate, --effort low only, three short-answer programmatically-graded families, no multi-turn work, and pricing sensitive to scheduled changes. Dollars are derived from token counts against a checked-in price table, not billed.

Every figure recomputes offline from committed raw per-call data in seconds, with no API key. Nothing is hand-typed into the results — clone it and run make bench-replay.

Stack

Python, uv, FastAPI, Postgres (table-as-queue, SELECT FOR UPDATE SKIP LOCKED), OpenTelemetry. No Kubernetes, no Terraform, no queue broker.