Observability & logging
Why the convention matters more than the tool
Every observability product arrived with its own tracing format, and choosing one used to mean rewriting instrumentation to leave it. The OpenTelemetry GenAI semantic conventions standardise the vocabulary, span names, model attributes, token counts, tool calls, agent steps, so the same emitted telemetry can be read by Langfuse, Phoenix, Tempo, Jaeger or a cloud vendor's backend without touching application code. The conventions are still evolving, which is an argument for emitting them through a shared module rather than by hand.
| Layer | What is emitted | Why |
|---|---|---|
| Agent span | One root span per run: use case, tenant, agent name, run contract, outcome, total cost. | The unit a human reviews and a client is billed for. |
| Model call spans | Provider, model ID, temperature, input tokens, output tokens, cached tokens, latency, finish reason. | Cost and latency attribution to the exact call, not the whole run. |
| Tool call spans | Tool name, namespaced source, arguments schema, result size, error class, retry count. | Most agent failures are tool failures wearing a model's voice. |
| Retrieval spans | Store, query, filters, number of chunks, scores, which store answered. | The difference between a hallucination and a retrieval miss is visible only here. |
| Evaluation scores | Judge scores attached to the trace they grade, with rubric and judge versions. | Quality becomes queryable next to cost and latency instead of living in a separate spreadsheet. |
| Structured run log | One JSON line per run mirroring the root span, written regardless of backend availability. | Telemetry pipelines fail; the ledger must not. |
Content capture and the privacy line
- Traces are content-blind by default. Prompt and completion bodies are off unless explicitly enabled for a named environment, and never enabled for a tenant whose data-rights clause forbids it.
- Redact at the SDK boundary, not at the dashboard. Anything that leaves the process has already been filtered; a redaction rule that lives in the viewer is not a control.
- Tenant ID on every span, always. It is the attribute that makes isolation auditable and per-client cost real.
- Sample deliberately. Full capture in development and on evaluation runs; head or tail sampling in production, with errors always kept.
- Traces are business-blind. The harness records that a run happened, what it cost and whether it passed, not what the deal was worth.
What the dashboards must answer without a code change
Backend choice is an environment-layer decision, not an application one: Langfuse for LLM-native tracing with prompt management and scoring built in, Phoenix where evaluation and experimentation sit closest to the traces, Tempo or Jaeger where the AI stack must sit inside an existing SRE estate. All three read the same emitted conventions, which is the entire point.