Engineering Standards › Project skeleton standard

Project skeleton standard

Every AI project starts from the same skeleton. Not a suggestion, a stamp. The point is not tidiness: it is that a reviewer, or a new joiner three time zones away, can open any repository in the company and know within thirty seconds where the models are declared, where the evals live, what the agent is allowed to do and what the last run cost. A repository that cannot be diagnosed at a glance cannot be managed at a distance.

The root standard

Path	Purpose	The rule that makes it work
models/registry.*	The single declaration of every model the project may call: logical role, provider, model ID, tier, context window, price per million in and out.	Names and metadata only. No API keys, no client objects, no call code. Every call site resolves a role (drafter, judge, extractor) and never a raw model string, so a model swap is one line in one file.
config/	Environment and run configuration, layered: defaults, environment overlay, run overrides.	Values from the environment, secrets from the vault, variable names in .env.example and values never. Config is read once at startup and passed down; nothing reaches into the environment mid-call.
prompts/	Versioned prompt and system-message assets, one file per role, with a version header.	Prompts are owned artifacts under review, not string literals buried in application code. A prompt change is a diff, and a diff is reviewable.
agents/	One definition per agent: objective, output format, tool and source guidance, task boundaries.	Four things and no more. Domain rules live in the rule library; orchestration state lives in the orchestrator.
skills/	The S·S·R·A folders, one per use case.	A new use case adds one folder here and touches nothing else.
tools/	Tool and MCP server implementations, namespaced by source.	Tools move and shape data. A tool that decides, drafts or formats is the Smart Pipe antipattern.
evals/	datasets/ golden sets · rubrics/ judge prompts · judge.py the LLM-as-judge runner · baselines/ stored scores.	Versioned in the repo beside the code it grades. A prompt change and its eval result travel in the same pull request.
observability/	Tracing setup, span helpers, redaction filters, cost accounting.	Instrumentation is a module, not a scattering of log lines. One import, one initialiser, uniform span names.
logs/	Structured run records: run ID, tenant, use case, model, tokens in and out, cached tokens, latency, cost, outcome.	Structured records only, one JSON object per line. Free-text logging is for humans debugging; structured logging is what the cost and quality dashboards read.
tests/	unit/ · integration/ · e2e/ for the deterministic code.	Evals do not replace tests. Deterministic code gets ordinary tests; only the judgment surface gets evals.
infra/	Infrastructure as code, one module set per provider.	Nothing is clicked in a console. If it is not in here, it does not exist in an environment anyone else can reproduce.
CLAUDE.md	The root constitution: coding standards, repo layout, build and test commands, review rules.	Short, universal, one per repository. If a rule applies to only some sessions, it is not a constitution rule.
docs/	Architecture, API reference, onboarding, decision records.	An ADR for every architectural choice, so the next person inherits the reasoning and not only the result.

The tree, in full

This is the stamp. A new AI application is generated from it and starts life with authentication, persistence, model routing, prompt versioning, guardrails, evals, tracing and CI gates already wired, because every one of those is far harder to retrofit than to inherit. Directories are omitted only when the application genuinely has no such concern, and that omission is recorded in an ADR.

ai-service/                          ONE deployable AI application
  CLAUDE.md                          root constitution: standards, layout, build & test commands, review rules
  AGENTS.md                          symlink to CLAUDE.md, so non-Claude agents read the same file
  README.md                          what it does, how to run it, who owns it
  Makefile                           setup | lint | test | eval | run | deploy, one verb each, no exceptions
  pyproject.toml                     src-layout package; absolute imports only, never relative
  .env.example                       variable NAMES only; values live in the vault
  Dockerfile  docker-compose.yml     the same image CI builds is the image production runs

  .claude/                           the agent-facing half of the repo
    settings.json                    shared configuration, checked in
    settings.local.json              personal overrides, gitignored, never committed
    commands/                        run-use-case.md  validate.md  eval.md  regenerate.md  promote.md
    hooks/                           format | lint | schema-check | tenant-isolation | eval-regression
    agents/                          orchestrator.yml + one .yml per business agent

  skills/                            S.S.R.A, one folder per use case, nothing else
    <use-case>/
      SKILL.md                       frontmatter (name = folder name) + the procedure
      scripts/                       deterministic steps; source never enters context
      references/                    loaded on demand, one file at a time
      assets/                        output templates and static files

  src/app/
    main.py                          composition root: load settings, wire DI, mount routers. No logic here.

    core/                            cross-cutting concerns; imports nothing from feature packages
      config.py                      typed settings (Pydantic Settings); fails fast at startup, never mid-request
      logging.py                     structured JSON logger; run_id and tenant_id on every single record
      telemetry.py                   OTel tracer and meter, GenAI semantic conventions, redaction filter
      security.py                    hashing, token issue and verify, the tenant context variable
      exceptions.py                  the one exception hierarchy every layer raises into
      middleware.py                  request ID, tenant resolution, timing, global error handler
      constants.py  pagination.py

    api/                             HTTP mechanics only: parse, validate, delegate, return
      deps.py                        injected dependencies: current_user, tenant, db session, services
      v1/
        router.py                    aggregates this version; versions are additive, never edited in place
        routes/                      auth.py  chat.py  runs.py  documents.py  admin.py  health.py
      schemas/                       request and response contracts (Pydantic). ORM objects never cross this line.

    auth/
      router.py  service.py  dependencies.py
      models.py                      user, tenant, role, api_key
      permissions.py                 the RBAC matrix: role -> permitted use cases, tools and data scopes
      providers/                     oidc.py  api_key.py  service_account.py

    db/
      session.py                     engine, session factory, async context manager
      base.py                        declarative base and naming convention
      models/                        ORM models, one module per aggregate
      repositories/                  the ONLY place queries are written; one repository per aggregate
      migrations/                    alembic; every schema change is a migration, never a hand edit
      seeds/                         reference data, checked in

    llm/                             the portability boundary; everything above is provider-blind
      registry.py                    role -> provider, model_id, tier, context window, price in and out
      router.py                      picks the tier per job class and enforces the cost ceiling
      providers/                     anthropic.py  bedrock.py  openai.py, thin adapters, one interface
      cache.py                       prompt-cache breakpoints, prefix hashing, hit-rate metrics
      budget.py                      per-run token budget; over budget is a kill, not a warning
      schemas.py                     the structured-output models every call is validated against

    prompts/
      system/                        one file per role, version header at the top of each
      templates/                     static prefix first, volatile suffix last, the cache depends on it
      registry.py                    name and version -> rendered prompt, hashed for cache stability

    agents/
      base.py                        the loop: objective, output format, tool guidance, task boundaries
      orchestrator.py                deterministic; routes, gates, never authors
      finance_agent.py  deals_agent.py  ppm_agent.py
      subagents/                     internal workers returning bounded summary contracts

    tools/                           one module per source, namespaced, high-signal returns
      airtable.py  sharepoint.py  vector_search.py  graph_query.py  render_deck.py
      registry.py                    the only path by which a tool becomes callable

    retrieval/
      vector_store.py                per-tenant namespaces, never a shared index
      graph_store.py                 ontology queries only, against the pinned release
      chunking.py  embedding.py  rerank.py
      context_pack.py                just-in-time assembly and the compaction policy

    domain/                          deterministic business rules: computed once, cited everywhere
      rules/  calculators/  validators/

    workflows/                       deterministic multi-step orchestration, not agents
      ic_memo.py  ingestion.py

    guardrails/
      input_filters.py               injection screening, PII detection on the way in
      output_filters.py              schema validation, citation check, PII on the way out
      policies.yaml                  declarative, versioned, reviewed like code

    workers/
      queue.py  tasks.py  scheduler.py    long runs, batch jobs, checkpoint and resume

  evals/                             graded beside the code it grades
    datasets/                        golden/  adversarial/  edge/  replays/ , versioned JSONL
    rubrics/                         judge prompts, versioned alongside the datasets
    judge.py                         cross-family judge, order rotation, rationale required
    run_eval.py                      the CI entry point: writes scores, returns an exit code
    baselines/                       the stored scores the gate compares against

  tests/
    unit/  integration/  e2e/
    fixtures/                        recorded model responses, so tests are deterministic and free

  infra/
    terraform/                       one module set per provider; nothing is clicked in a console
    k8s/                             manifests or charts
    ci/                              the gates: lint | test | eval | tenant-isolation | promote

  scripts/                           setup.sh  ingest.py  reindex.py  backfill.py
  docs/
    architecture.md  onboarding.md  api-reference.md
    adr/                             one record per architectural decision, numbered, never deleted

The boundaries that make it work

A folder tree is only a convention until something enforces it. These six rules are what a reviewer checks and, where the check is mechanical, what a hook fails the build on.

Dependencies point one way. api may call agents, workflows and domain; those may call llm, tools, retrieval and db.repositories; everything may call core. Nothing calls upward and nothing skips repositories to reach the database. An import-linter rule enforces this, because a layering convention nobody can violate is the only kind that survives a deadline.
Routes are thin, services are testable, repositories own the data. A route parses, validates, delegates and returns. If business logic can only be exercised by starting a web server, it is in the wrong file.
Two model vocabularies, never confused. db/models is persistence, api/schemas is the wire contract, llm/schemas is the structured-output contract. Returning an ORM object from a route is how internal columns end up in a client's browser.
Model identity lives in exactly one file. Every call site resolves a role, drafter, judge, extractor, router, against llm/registry.py. A grep for a raw model string anywhere else is a build failure, which is what makes a model migration a one-line change instead of an archaeology project.
Secrets have exactly one shape. Names in .env.example, values in the vault, resolution in core/config.py, and nothing reads the environment directly after startup.
Tenant is carried, not looked up. Middleware resolves it once into a context variable; repositories, stores and traces all read it from there. Isolation is then a property of the framework rather than a discipline each developer has to remember, and the isolation test in CI proves it on every commit.

Constitution discipline

The most common failure found in review is not a missing file; it is a constitution carrying work that belongs elsewhere.

One constitution per repository, not one per use case. A tree with a CLAUDE.md in every folder is a tree where nobody knows which rules are in force. Use-case procedure belongs in that use case's SKILL.md.
Business logic in a constitution is a defect. It is universal conventions only: how to build, how to test, how to review, where things live.
Link, never paste. Long reference material is referenced from the constitution and loaded on demand.
Keep it short and refine it like a prompt. It is read on every session; every line is paid for on every session.
Hand-write it. Machine-generated constitutions read as generic advice the agent already follows, "write clean code", "follow best practices", and measurably degrade task performance. Write only what the agent could not infer from the codebase itself.
Client constitutions are generated, never edited. They regenerate from the binding whenever plane values change; a hand edit is silently overwritten and is therefore a bug in waiting.

The engineering posture underneath

The skeleton encodes a position that the published production-agent guidance and our own build history agree on: reliable agent systems are mostly ordinary deterministic software with small, tightly controlled model calls at the points that genuinely need judgment.

Own the parts that fail. Own your prompts, own your context window, own your control flow. Abstractions that hide any of the three make production incidents undiagnosable. A framework is acceptable only if those three surfaces stay inspectable.

Structured output is the contract. The model returns JSON against a schema; the schema is validated before anything downstream runs. Free text crossing a module boundary is an outage waiting for volume.

Small, focused agents beat one large one. Narrow scope, bounded context, explicit hand-offs. The failure mode of the large agent is that nobody can say why it did what it did.

Stateless where possible, checkpointed where not. Long runs summarise completed phases into external memory and resume; they never restart from zero.

Humans are reached by tool call. Escalation and approval are explicit calls with typed payloads, not an agent hoping someone reads a log.

Errors are compacted into context, not looped on. A failed tool call returns a compact, actionable message; retry policy is code, not vibes.

← Repository skeleton Evals & LLM-as-judge →