Engineering Standards › Evals & LLM-as-judge
Evals & LLM-as-judge
Build the eval before you build the thing it grades. The eval suite is what converts
"this feels better" into a number a reviewer can act on, and it is the only mechanism that makes distributed work safe: a contributor three
time zones away cannot argue with a failing gate. Written after the fact, it encodes whatever the system already does, including its bugs.
The golden set
Coverage of distinct failure modes matters far more than raw count. A hundred varied cases beat a thousand near-duplicates.
| Bucket | What goes in it | Why it exists |
|---|---|---|
| Production sample | A stratified sample of real traffic, PII removed, mirroring the live query distribution. | Without it the suite grades a world the system does not live in. |
| Adversarial | Prompt injection attempts, contradictory instructions, out-of-scope asks, missing-source questions. | Guardrails that are never tested are decoration. |
| Edge cases | Deliberately constructed boundaries: empty documents, conflicting figures, unusual entity structures, the largest realistic input. | These are where quality actually degrades first. |
| Failure replays | Every defect that ever shipped, captured as a case the moment it is diagnosed. | The one bucket that guarantees a bug is fixed once rather than repeatedly. |
- Start at twenty to fifty reviewed cases. That already catches gross regressions and is small enough that it actually gets built. Grow toward a few hundred as the system matures.
- Keep a fast subset for every pull request and run the full suite before release, so the gate is quick enough that nobody routes around it.
- Version the dataset in the repository and treat a dataset change like a code change: reviewed, diffed, attributable.
- Hold out a partition that never runs in CI. If gated scores and holdout scores diverge, the system has been tuned to the test rather than to the task.
- Refresh monthly from production failures. A suite that never gains new cases stops catching new regressions.
LLM as judge, run properly
A calibrated judge tracks human reviewers closely enough to be trusted at volume, and costs a fraction of human review. An uncalibrated one produces confident numbers that mean nothing. The difference is entirely process.
| Control | The practice | The failure it prevents |
|---|---|---|
| Calibrate before trusting | Score a labelled sample by hand, then measure judge agreement against it. Only promote the judge once agreement clears the threshold you set in advance. | Shipping a judge whose scores correlate with nothing. |
| Cross-family judging | The judge comes from a different model family than the system under test. | Self-preference: models score their own family's output higher. |
| Order rotation | In pairwise comparisons, run both orders and average. | Position bias, which is large, consistent, and model-specific in direction. |
| Decompose the rubric | Many small binary or low-cardinality checks rather than one holistic score out of ten. | Vague single scores that drift and cannot be diagnosed when they fall. |
| Demand rationales | The judge returns reasoning alongside the score. | Silent rubric misinterpretation; a judge that cannot explain itself cannot be debugged. |
| Normalise for length | Penalise or normalise verbosity explicitly in the rubric. | Verbosity bias, where longer answers score higher at equal quality. |
| Pin the judge contract | Version the triple of judge model, rubric version and prompt template together; bump it deliberately. | Score movements caused by a silent judge change being read as product regressions. |
| Keep humans in the loop | A standing sample is reviewed by people, always. | The judge and the system drifting together into agreement about the wrong thing. |
What we score
Factual accuracy. Are the stated numbers and claims correct against the source?
Citation accuracy. Does every claim point at a source that actually supports it?
Completeness. Were all required sections and required data points produced?
Source quality. Did it draw on the authoritative source rather than the convenient one?
Tool efficiency. Calls made against calls needed, the cost dimension of quality.
End state, not steps. Judge whether the correct final state was reached. There are many valid routes; grading the route punishes good ones.
The three tiers
- Offline, against the golden set. Run before every release, per component, against a stored baseline. Answers: did we break something that used to work?
- The CI gate. Automated on every pull request, per-bucket thresholds, merge blocked on regression beyond tolerance. Answers: is this specific change safe to ship? Set thresholds with headroom, tight enough to catch degradation, loose enough that ordinary run-to-run variance does not cry wolf.
- Online, on live traffic. Continuous sampling, drift detection, alerting when a quality metric falls below baseline for a sustained window. Answers: is it working right now, for real users?