Engineering Standards › Token & cost engineering
Token & cost engineering
Token cost is an architectural property, not a monthly surprise. A single internal
workstream reaching roughly nine hundred dollars is not a billing problem; it is the absence of a caching and routing discipline that should
have been in the harness from the first commit. If we cannot make cost repeatable across five people, we cannot sell cost optimisation to a
company of five thousand.
Prompt caching, first and hardest
This is the highest-leverage change available to most projects and typically takes under an hour to implement. Cache reads are charged at roughly a tenth of the base input price, and cache writes at a modest premium over it; the published guidance reports cost reductions up to about ninety percent and latency reductions up to about eighty-five percent on long, stable prefixes.
- Order the prompt static-first. Tools, then system instructions, then reference material, then conversation, then the volatile user turn. The cache matches a prefix; anything dynamic placed early destroys everything after it.
- Mark the breakpoint explicitly at the end of the stable block, and let it advance with the conversation so that earlier turns become cache reads.
- Serialise deterministically. Most mysterious cache-hit collapses are a reordered JSON key, an added whitespace, or an injected timestamp inside what was meant to be the stable prefix. Snapshot the serialised prefix in a test and fail the build when it changes unintentionally.
- Expect the first turn to miss. Caching pays from the second interaction; pre-warm where first-token latency is user-visible.
- Instrument the hit rate per route and treat a fall in it as an incident, not a curiosity. Published production accounts show total spend falling by well over half purely by raising hit rate from single digits to the eighties.
- Verify it is on before optimising anything else. The architecture review question is literal: does prompt caching appear as a named component in the infrastructure, environment or observability layer? If not, it is added there, not bolted onto one use case.
The rest of the ladder
| Technique | What it does | Where it lives |
|---|---|---|
| Model routing | Sends each job class to the cheapest model that clears its quality bar; escalates only on failure or low confidence. Most steps in a document pipeline are extraction and formatting, not reasoning. | Intelligence layer, behind the router; per-use-case tier preference is a plane value. |
| Token budgets per run | A ceiling declared in the run contract. Over budget is a kill, not a warning, because the runaway loop is the expensive failure mode. | Environment layer; the harness enforces it. |
| Context compaction | Summarising completed phases into external memory instead of carrying the whole transcript forward. Context bloat is the quiet multiplier on every subsequent call in a long session. | Session and memory machinery. |
| Just-in-time retrieval | Load the reference file the step needs, not the corpus that might be needed. This is exactly what the references/ directory is for. | Skill authoring. |
| Parallel tool calls | Fanning independent calls out concurrently; the published research reports research time cut by up to ninety percent on complex queries. | Orchestrator, within the dependency graph. |
| Batch processing | Non-interactive work submitted asynchronously at a substantial discount, stacking with caching. | Any scheduled or overnight job. |
| Effort scaling | A simple lookup gets one worker and a handful of calls; a broad comparison gets several workers. Declared in the run contract rather than left to the model's discretion. | Run contracts, and the reason multi-agent runs, which cost roughly fifteen times a single chat, are dispatched deliberately. |
| Small models for narrow jobs | Classification, extraction, routing and reranking rarely need a frontier model. | Intelligence layer; the router chooses. |
Making it repeatable
The commercial argument is not that a given number was too high; it is that we can state, for any project, what it should cost and why. That requires the measurement to be part of the platform rather than an investigation someone runs after the invoice arrives.
Every run carries its cost. Tokens in, tokens out, cached tokens and computed spend are attributes on the root span, always.
Cost ceilings are plane values. The client sets them; the router enforces them; breaching one raises an alert rather than an invoice.
Optimisations ship as versioned config. A tuning change goes through CI like any other change. Silent drift is a defect.
The cost review is a standing agenda item with the same weight as the quality review, using the same dashboards.