Model routing on a publisher's budget

A publisher's AI budget is not a rounding error. When agents are drafting summaries, tagging archives, generating brand kits and reviewing code across a dozen properties, the difference between a considered routing policy and “always use the best model” is the difference between a line item and a problem.

What follows is the policy I actually run, arrived at by overspending first.

Three tiers, one rule each

Local tier

Runs on hardware I already own. Gets anything high-volume, low-stakes and privacy-sensitive: OCR cleanup, tagging, deduplication, first-pass classification of archive material, transcript segmentation. Marginal cost is electricity. The rule: if a human would not read the output directly, it goes local.

Cheap hosted tier

Small, fast hosted models. Gets structured extraction, short rewrites, metadata generation, alt text, routine summarisation. The rule: if the task has a schema, a cheap model with a good prompt beats an expensive model with a vague one.

Frontier tier

The expensive models. Gets code review, architecture decisions, editorial judgement, anything customer- facing, and anything I will publish under a masthead. The rule: if being wrong is embarrassing or expensive, pay for it.

Most cost blowouts are not caused by expensive models. They are caused by expensive models doing cheap work in a loop.

The routing table

route(task):
  if task.privacy == "sensitive"        -> local
  if task.volume > 1000/day             -> local
  if task.output_schema is not None     -> cheap
  if task.reviewed_by_human is False    -> cheap
  if task.published or task.is_code     -> frontier
  else                                  -> cheap, escalate on low confidence

The last line matters more than the rest. Escalation on low confidence means the cheap tier handles the long tail and the frontier tier only sees the hard cases. That single pattern cut spend more than any model swap.

What I got wrong

Optimising the wrong axis. I spent a week shaving token counts on a task that ran twice a week. Meanwhile a tagging job was quietly calling a frontier model forty thousand times a month. Always sort by calls × price, never by price.

Trusting caching to save me. Prompt caching helps when prompts are stable. Mine were being regenerated with a timestamp in them. Cache hit rate: near zero. Check, do not assume.

Treating retries as free. A retry on a frontier model is a full-price call. Retry on the cheap tier, escalate once, then fail loudly.

The number that matters

I track cost per published item — per article, per brand kit, per invoice reconciled. Not tokens, not spend per month. Cost per published item is the only figure that tells me whether the AI layer is earning its place in a publishing business, and it is the figure I would put on the wall if I were doing this again from the start.