Engineering Standards › Model customisation
Model customisation
Fine-tuning is the last lever, not the first. The order is prompt, then context, then
retrieval, then routing, then distillation or fine-tuning. Most quality problems that look like model problems are retrieval problems, and
most that survive retrieval are prompt problems. When customisation genuinely is the answer, it happens inside the Intelligence layer and
changes nothing above it.
When customisation is actually warranted
House voice and format at volume. Where the output must match a rigid structure every time and prompt-based enforcement keeps drifting.
A narrow, high-frequency task. Where a small tuned model matches a frontier model's quality at a fraction of the cost and latency.
Domain vocabulary. Where a specialist register is not reliably reproduced by instruction alone.
Not for facts. Changing knowledge belongs in retrieval and the ontology; a tuned model with stale facts is a liability that has to be retrained to correct.
Not for reasoning. If the base model cannot do the reasoning, tuning on a few thousand examples will not teach it.
Not before evals exist. Without a golden set there is no way to prove the tuned model is better, and every subsequent decision becomes a matter of opinion.
Routes we would actually use
| Route | What it gives | Trade-off |
|---|---|---|
| Managed supervised fine-tuning Bedrock, Vertex, Azure AI Foundry | Labelled input-output pairs against a supported base model, inside the cloud account where the data already sits. Billed on tokens processed times epochs, plus model storage. | Constrained to supported base models; the customised model needs provisioned capacity in most managed offerings. |
| Reinforcement fine-tuning Bedrock, open-weight models | Reward functions instead of labelled pairs, using rule-based graders or an AI judge, with a small prompt set rather than a large labelled corpus. Now extended to open-weight families with OpenAI-compatible APIs, with intermediate checkpoints exposed for evaluation. | Reward design is the whole job, and a badly specified reward is a confidently wrong model. This is where our eval rubrics earn their second use. |
| LoRA and QLoRA adapters SageMaker, self-managed, Fireworks | Base weights frozen, small trainable adapters injected. Cheap to train, cheap to store, and multiple adapters can share one GPU with swapping per request, the practical route when several tuned variants are needed. | Requires more of the training stack to be understood and owned. |
| Hosted tuning platforms Fireworks AI and similar | Supervised fine-tuning from JSONL in the standard chat-completion format, and adapters trained elsewhere can be uploaded for serving. Fast path from dataset to a served endpoint. | A second vendor boundary and a second place data can rest; both need the data-rights clause checked before anything is uploaded. |
| Custom model import Bedrock | Bring weights customised anywhere, supported open-weight architectures, and serve them managed, without running the serving stack. | Architecture support is a fixed list; check it before choosing a base model, not after training one. |
The discipline around any of them
- Baseline first. Record the prompt-only score on the golden set. A tuned model that does not beat it is a cost with no benefit, and without the baseline nobody can tell.
- Dataset provenance is checked before training, not after. Which tenant, under which clause, anonymised how. Client data crossing into a shared tuned model without a rights check is the failure that ends a relationship.
- Version the model like code. Dataset version, base model, hyperparameters and resulting model ID recorded together; the registry entry points at that record.
- Evaluate on the holdout, not the training set, and against the same rubric used in production.
- Keep the swap reversible. The tuned model enters through the router as one more role binding, so rollback is a config change and never a rebuild.
- Re-evaluate on every base-model release. The frontier moves fast enough that a tuning effort can be obsoleted by a general model within a quarter; the eval suite is what tells you when.
The open-weight route has a second use worth stating plainly: it is the cleanest way to measure what the knowledge layer actually contributes. With a frontier model it is hard to separate the model's own competence from the ontology's influence. Load the same ontology into an open-weight model and the difference becomes measurable rather than arguable.