The stack uses a per-service folder layout under services/<name>/. Each service family (one or more co-lifecycled containers) owns:

• service.yml — manifest: env vars, source variants, image refs, dependencies, plus per-source bootstrapper runtime data under runtime_sc: / runtime_adaptive: / runtime_deps:
• compose.yml — Docker Compose fragment for that family's containers
• optional subdirectories holding Dockerfiles, init scripts, source code, configuration files, snapshots, etc. — see the Subdirectory naming convention below

The thin top-level docker-compose.yml merges fragments via Compose's native include: directive (requires Compose ≥ v2.20.3; v2.26+ recommended).

Each step links to its deep-dive section.

• Study the candidate service's upstream docs (license, default port, API shape, runtime deps) → Pre-flight
• Pick a folder flavor → Decision 1
• Pick a category → Decision 2
• Pick your sources → Decision 3
• Write services/<name>/service.yml → Mechanics
• Write services/<name>/compose.yml (only if folder flavor = container) → Mechanics
• Add the include: line to docker-compose.yml (only if you wrote a compose fragment)
• Register CLI key in source_mapping → Mechanics — source_override_manager registration. Without this the wizard silently skips your service.
• Add the new folder to the relevant track(s) in bootstrapper/tracks.yml (source-configurable services only). A configurable service absent from a named track's services: list is force-disabled (*_SOURCE=disabled) there — it only runs under --track all. Always-on infra and the always-prompted LLM/Prometheus/Grafana tier are exempt.
• Add the row's display name to EXPECTED_DISCOVERED in bootstrapper/tests/test_wizard_app_discovery.py (source-configurable services only). The discovery-count test fails until you do.
• runtime_sc key differs from the folder name (for example ray-head in a multi-container family): add the mapping to _FAMILY_KEY_ALIASES in bootstrapper/tracks.py. Otherwise the track filter does not match it.
• Add the new containers to the fragment baseline → Byte-equivalence
• Run the root-safe regen, lint, and required-check checklist → After you save the files, then CI gates
• Update audit-script allowlists if your service has hard deps → Audit-script + CI implications
• Commit and push. CI gates the change with four required jobs: manifest-lint+pytest, compose-equivalence+permutation matrix, docs-drift+audit-scripts, and build-validation.

If you're new to this codebase, read Decisions 1–6 in sequence; the Qdrant worked example illustrates each one.

Worked example throughout: Adding Qdrant, a self-hosted vector database. (Qdrant is real software but NOT currently in the stack — Weaviate is our vector DB. Used here purely as an instructional example, not a proposal.)

Read the Pre-flight study (§4), then Decisions 1–6 (§5–§10). The Qdrant example runs through all of them. §12 lists the regen and lint commands; skipping one fails the byte-equivalence test or the docs-drift gate.

Read the candidate service's upstream docs before you write a manifest. Every later decision depends on these facts. A wrong port or API assumption carries through to category, sources, compose and Kong.

Once you understand the candidate, scan our existing service manifests (run grep -l "^data_flow:" services/*/service.yml) to identify integration points:

• Upstream callers (who in our stack would call this new service). Run grep -l "^data_flow:" services/*/service.yml and skim each service's data_flow.calls list. Which existing services would benefit from calling this new one? For a new vector DB: Backend, n8n, JupyterHub, possibly Hermes Agent. Add these edges to those EXISTING manifests' data_flow.calls lists, NOT to your new service's depends_on. (See Decision 5 for why data_flow.calls is separate from depends_on.)
• Downstream callees (what this service calls). Does the candidate make outbound calls to anything we already run? Most app-tier services touch Supabase (auth/storage), LiteLLM (LLM access), and Redis (caching). These would be entries in YOUR new service's data_flow.calls.
• Source-variant precedents. Find the closest existing service that ships similar source variants and use its manifest as a template:
  • New vector DB → services/weaviate/service.yml
  • New LLM gateway / engine → services/litellm/, services/ollama/
  • New STT/TTS engine → services/parakeet/, services/speaches/, plus the aggregator pattern in services/stt-provider/ + services/tts-provider/
  • New app-tier UI → services/open-webui/, services/jupyterhub/
  • New cloud-API toggle (not a container) → services/cloud-providers/service.yml

Integration discovery for Qdrant. Backend and n8n already call Weaviate; Qdrant would be a sibling vector-DB option for users who prefer it. JupyterHub users running RAG experiments might want a second vector store for A/B comparisons. No service in our stack would be a downstream caller — Qdrant is a leaf vector store. → On Qdrant's manifest, data_flow.calls: []. Integration entries appear in services/{backend,n8n,jupyterhub}/service.yml's data_flow.calls lists (added in those manifests, not in Qdrant's).

Source-variant precedent. Weaviate is the closest sibling — same category (data), same role (vector DB), similar source-variant shape. Open services/weaviate/service.yml side-by-side while drafting services/qdrant/service.yml.

With pre-flight complete, the six decisions become near-mechanical — proceed to Decision 1.

Three legitimate flavors of folder live under services/. Pick the right one before writing anything else.

Flowchart:

1. Does it run as a container with its own image? → container.
2. Does it own env vars / SOURCE toggles but with no compose fragment of its own? → virtual. Set virtual: true; omit compose.yml. The validator enforces this.
3. Is it documentation-only — aggregating other manifests under one user-facing role? → doc-only. No service.yml, no compose.yml, just README.md + architecture.svg / architecture.html.

Worked example — Qdrant: Qdrant ships as a container image (qdrant/qdrant:v1.12.0), exposes a real HTTP API, and has its own env vars → container flavor.

Common mistakes:

• Adding a virtual manifest with a compose fragment — the manifest validator rejects it (unexpected_fragment). Either remove compose.yml (if no container runs) or unset virtual: true and keep the compose fragment (container flavor).
• Adding a doc-only folder when the role has env vars to manage — use a virtual manifest instead.

Every manifest declares one of six categories. The category drives two things: the wizard block your row renders in, and the port-slot block your service draws from.

Effects of the category:

• Wizard placement. Categories render in fixed order (infra → data → llm → media → agents → apps). Within a category, services follow topological order (driven by depends_on.required).
• Port-slot block. Each category gets its own port-offset range — see Decision 4.
• Architecture-diagram clustering. The full-stack diagram at docs/diagrams/architecture.svg (hand-authored via the architecture-diagram skill) groups services by category band. Per-service architecture diagrams under services/<name>/architecture.svg use the same category palette but cluster the call graph instead.

Worked example — Qdrant: Qdrant is a vector database. Its closest siblings in the stack are Weaviate and Supabase (which are also data-tier). → category: data.

How to pick when you're unsure: find the most-similar existing service and use its category. If your service genuinely doesn't fit any of the six, that's a design conversation, not a category decision — open an issue first.

Every user-configurable service has an <SVC>_SOURCE env var. The wizard reads its values from the sources.options block in your manifest.

Standard option names (use these; don't invent new ones unless your service genuinely needs them):

Implications of your choices:

• Locked vs. user-choice. A service with only one source variant is "locked" — the wizard skips its prompt entirely. The _is_locked helper in bootstrapper/services/topology.py enforces this. Services like Backend, Kong, LiteLLM are locked because they're always-on.
• requires: per option. Use requires: [<ENV_VAR>] on a source option to declare prerequisite env vars (e.g. localhost typically requires <SVC>_LOCALHOST_PORT).
• profiles: per option. Mark each dev-only option profiles: [default], especially localhost, because host.docker.internal is useless on a remote host. An option without profiles: is offered in every profile, which suits container and disabled. Under --profile prod the wizard hides dev-only options, and bootstrapper/services/source_validator.py::validate_sources_for_profile rejects them.
  • The manifest-validator lint fails only when no prod-eligible option is left. Annotate each dev-only option yourself.
  • The matching Python predicate is option_in_profile in bootstrapper/services/manifests.py.
  • The profile bundles live in bootstrapper/profiles.yml: default and prod maps of sources, env and host_bind_ip that apply_profile_overrides applies at start. Consumers override fields with profile_overrides: in atlas.consumer.yml (Reusing Atlas §6.1).
• <SVC>_LOCALHOST_PORT is the single source of truth. If you offer localhost, declare <SVC>_LOCALHOST_PORT (an integer string, defaulting to the upstream host port). The URL is derived as http://host.docker.internal:${<SVC>_LOCALHOST_PORT:-<default>}. The in-container consumers (runtime_sc.<svc>.localhost.environment) and the Kong route generator (bootstrapper/utils/kong_config_generator.py) must read the same var. The wizard shows an inline port box on the localhost row. See also Common gotchas.
• runtime_sc slice per source. Every source variant declared in sources.options must have a matching runtime_sc.<key>.<source> slice. By convention each slice sets scale, environment, deploy and extra_hosts. The manifest validator reports a missing slice as runtime_sc_missing_variant, so a declared option cannot silently scale to 0.
• auto_prefer: (optional). This enables <SVC>_SOURCE: auto in atlas.consumer.yml. The resolver picks the first entry whose requires_capability (apple_silicon, nvidia_gpu, host_ollama; probed by bootstrapper/services/host_capabilities.py) holds on the host and whose option the active profile offers.
  • Each id must be in options[].id. The last entry must have no requires_capability; it is the terminal fallback, usually the CPU container. The lint (_check_auto_prefer_integrity) enforces both rules.
  • Without auto_prefer, auto resolves to default with a warning.
  • Reference shapes: services/comfyui/service.yml, services/ollama/service.yml.

Worked example — Qdrant: Most users won't already run Qdrant locally, so container is the primary path. We offer three variants — external is deliberately omitted per the stack-wide moratorium noted above:

• container — default, scale=1
• localhost — QDRANT_LOCALHOST_PORT defaults to "6333" (Qdrant's standard host port); the URL is derived at compose-render time as http://host.docker.internal:6333
• disabled — scale=0

The bootstrapper auto-assigns ports. You do not pick a port.

How it works:

• BASE_PORT (default 63000) is the bottom of the port range. Users can override with ./start.sh --base-port 64000.

• CATEGORY_SLOTS in bootstrapper/services/topology.py assigns each category an (offset, block_size) tuple. Current values:

  Category	Offset	Block size	Resolved range with default BASE_PORT=63000
  infra	0	10	63000-63009
  data	10	30	63010-63039
  llm	40	10	63040-63049
  media	50	20	63050-63069
  agents	70	20	63070-63089
  apps	90	20	63090-63109

• Within each category block, services consume slots in topological order (driven by depends_on.required — see Decision 5). Multi-port services (e.g. Supabase's 8 port vars, Weaviate's HTTP + gRPC pair, MinIO's API + Console pair) get a contiguous run.

• A category-overflow lint trips if you blow past your block. Fixes: move manifests to a different category (rare), or extend the block size in CATEGORY_SLOTS (also rare — coordinate with maintainers).

How to declare a port:

In your env: block, declare name: <SVC>_PORT with NO default: line. Convention is a single-line comment:

env:
  - name: QDRANT_PORT
    # default removed — computed by services/topology.py slot allocator

The services/env_assembler.py regen step emits the resolved port into .env.example. Never hand-edit .env.example — it's a generated artifact (byte-equivalence-tested in CI).

Worked example — Qdrant: Qdrant declares QDRANT_PORT with no default. Topology slots it into the data block at the next free offset. Its place in the topo sort relative to its siblings sets that offset (Supabase microservices, Redis, MinIO, Neo4j, Weaviate). The exact number is auto-resolved at every regen — don't pin it.

This is the most nuanced decision. The field depends_on.required in service.yml has two semantics that get conflated:

Runtime semantics (intended): "These services must be up before this one boots." Maps to the corresponding depends_on: block in your compose.yml fragment.

Display-ordering semantics (overloaded): The bootstrapper uses depends_on.required as the canonical-order backbone for the wizard's row order and the overview-box display. Within a category, services sort topologically; an edge from A → B means A appears AFTER B in the same category's wizard block.

Removing a non-runtime edge can still break the wizard. ollama lists litellm only to pin its row after LiteLLM. Removing that edge fails test_row_order_stability.py and shifts port slots in the LLM and media blocks.

• Use required for genuine bootstrap blockers AND for cross-category display-ordering pins.

• Comment any non-runtime edge inline as a display-ordering pin so readers don't try to "fix" it. Example from services/ollama/service.yml:

  depends_on:
    # `litellm` is listed here NOT because the ollama container calls
    # LiteLLM (the call direction is the opposite), but because the
    # bootstrapper uses `depends_on.required` as the canonical-order
    # backbone for the wizard / overview-box display. […]
    required:
      - supabase
      - litellm

• A future schema change may add a separate display_order: field that takes over the ordering job; until then, comment the intent.

• Don't list every service you call in required. Use data_flow.calls for that (it drives the architecture diagram and the per-service README's Dependencies & Integrations block — not topology).
• Don't list a virtual aggregate (globals, cloud-providers, tts-provider) in required as a boot dependency, because it has no container. You may list one as a commented display-order pin, as speaches and chatterbox do with tts-provider.
• Don't list optional deps that compose doesn't enforce. The optional list is documentation; if compose doesn't gate on it, it has no effect.
• Don't list a depends-on for a service that's source-replaceable (localhost, disabled, or a future authenticated remote mode). The audit script scripts/check-compose-source-deps.py enforces this; SOURCE-replaceable consumers should reach their target via endpoint env vars + runtime readiness checks, not via compose depends_on.

Worked example — Qdrant: Qdrant's only data-tier sibling that always runs is Supabase (the substrate the bootstrapper provisions before any other service starts). Listing it as a required dep pins Qdrant's position in the data block topologically. → required: [supabase], optional: [].

The data_flow.calls field is a runtime call graph that drives the architecture diagram and the per-service README's Dependencies & Integrations block. It is independent of depends_on. Use it to describe which services this one calls at runtime in the request path (excluding init-time bootstrap calls).

An entry is either a plain name, which declares a current edge, or an object that qualifies the edge:

data_flow:
  calls:
    - litellm                          # current: happens whenever both services run
    - target: mcp-servers
      status: optional                 # current | optional | planned
      condition: MCP_SERVERS_SOURCE=container
      evidence: services/jupyterhub/build/notebooks/15_mcp_clients.ipynb

• optional edges happen only under their condition, which is a SOURCE expression or a short free-text condition such as an operator setup step.
• planned edges are documented or intended but not wired yet.
• evidence is a repository path, path:line, or an https URL showing the call. The validator fails a repository path that does not exist.
• Each target appears once, whichever shape declares it.

The generated tables add a Status column, and the diagram draws a dashed pill, for any edge that is not current.

Most services express their per-source behavior with declarative runtime_sc YAML. The synthesizer in bootstrapper/services/sc_synthesizer.py concatenates all manifests' runtime_sc slices into the runtime service-config dict the bootstrapper consumes.

Declarative example (suffices for ~90% of services):

runtime_sc:
  qdrant:
    container:
      scale: 1
      environment:
        QDRANT_LOG_LEVEL: info
      deploy: {}
      extra_hosts: []
    localhost:
      scale: 0
      environment:
        QDRANT_ENDPOINT: http://host.docker.internal:${QDRANT_LOCALHOST_PORT:-6333}
      deploy: {}
      extra_hosts:
        - host.docker.internal:host-gateway
    disabled:
      scale: 0
      environment: {}
      deploy: {}
      extra_hosts: []

Write a helper in bootstrapper/services/service_config.py and wire it into generate_service_environment() ONLY when one of these is true:

1. Multi-input SOURCE dependencies. Your output depends on more than one <SVC>_SOURCE value. Example: _generate_stt_provider_config and _generate_tts_provider_config share a shared_env dict. STT runs first and writes SPEACHES_SCALE. TTS reads it, so Speaches is not scheduled twice when both roles pick a Speaches variant.
2. Derived / aggregated state. You need to compute env vars from a set of toggles. Example: _generate_cloud_providers_config reads the three CLOUD_*_SOURCE toggles and emits a per-provider enabled flag plus LITELLM_ENABLED_PROVIDERS (a comma-separated string).
3. Runtime-computed values. You need an env var whose value depends on another service's port, computed at runtime from BASE_PORT.

For everything else, stay declarative. Adding a hook means writing Python, adding a unit test for it, and giving future maintainers an extra place to read.

Worked example — Qdrant: Single SOURCE, no cross-service dependencies, no derived state. → No hook needed. The declarative runtime_sc covers every declared source.

Two adjacent fields that occasionally apply:

• runtime_adaptive — for services like backend that adapt their behavior based on which upstream services are enabled. Declares adapts_to: (a list of provider keys) and environment_adaptation: (env vars conditionally set when those providers are active). See services/backend/service.yml for the reference pattern.
• runtime_deps — launch-time dependency rules keyed by container.
  • requires names services this one cannot run without. If one is disabled, the bootstrapper prints error_message and auto-disables this service.
  • optional only prints which of the listed services are enabled ([INFO] <svc> will connect to: …).
  • conditional_requires adds rules that apply only when given env values are set.
  • Reference: services/n8n/service.yml.

Use these only if your service is genuinely adaptive. Today eight manifests declare runtime_adaptive (backend, comfyui, hermes, jupyterhub, lightrag, n8n, ollama, weaviate); backend is the most heavily adaptive and the canonical reference. Don't reach for these fields by default — start with declarative runtime_sc and only escalate when the adaptive behavior is non-trivial.

Every variable that generate_service_environment() writes to .env must have a consumer. Either a Compose fragment interpolates it (${VAR}), or it has a reviewed entry in FORWARDING_EXCEPTIONS (bootstrapper/tests/test_env_forwarding_contract.py).

• Valid exception reasons: host-only values, derivation inputs folded into another var, consumer-contract exports, container-less virtual families.
• An unconsumed variable fails that test; it never becomes a silent, inert .env entry.
• Remove an exception as soon as its variable gains a Compose consumer. The stale-entry guard checks both directions.

After making the six decisions, you write two files. Here's the full result for Qdrant. Line callouts (# ← Decision N) point back to the decision section that explains the choice.

# services/qdrant/service.yml — Qdrant vector database
name: qdrant
label: "Qdrant (vector database)"
category: data                                # ← Decision 2
docs: services/qdrant/README.md

support:                                      # required by tests (§22)
  tier: experimental
  evidence: "No qualification run cited yet."
  evidence_revision: v0.1.0                   # release tag or full 40-char commit id

capabilities:                                 # required by the schema
  - name: "Vector search over HTTP"
    status: supported                         # supported | partial | stubbed | not-supported
    verification: untested                    # tested | documented | untested
    note: "Single-node only; no cluster mode."

containers:
  - qdrant

images:
  - var: QDRANT_IMAGE
    default: "qdrant/qdrant:v1.12.0"
    container: qdrant

sources:                                      # ← Decision 3
  var: QDRANT_SOURCE
  default: disabled                           # off by default; user opts in
  options:
    - id: container
      label: "Container"
    - id: localhost
      label: "Host (existing Qdrant)"
      profiles: [default]                      # dev-only: hidden in the prod wizard, rejected by --profile prod
    - id: disabled
      label: "Disabled"
    # NOTE: `external` is deliberately omitted per the stack-wide moratorium
    # on `external` source variants — see Decision 3 above and the
    # `Removed (breaking)` section of docs/CHANGELOG.md.

env:
  - name: QDRANT_SOURCE
    default: disabled
  - name: QDRANT_PORT                         # ← Decision 4 (no default — auto-assigned)
    # default removed — computed by services/topology.py slot allocator
  - name: QDRANT_LOCALHOST_PORT
    default: "6333"
    description: "Host port for the qdrant-localhost source variant. URL is derived at compose-render time as http://host.docker.internal:6333. Same var consumed by Kong's qdrant.localhost route."
  - name: QDRANT_ENDPOINT
    auto_managed: true
  - name: QDRANT_SCALE
    auto_managed: true

depends_on:                                   # ← Decision 5
  required:
    - supabase
  optional: []

exports: []

rows:
  - display_name: "Qdrant"
    source_var: QDRANT_SOURCE
    port_var: QDRANT_PORT
    scale_var: QDRANT_SCALE
    alias: qdrant.localhost
    description: "Vector database (Qdrant)."
    localhost_endpoint_var: QDRANT_ENDPOINT
    localhost_port_var: QDRANT_LOCALHOST_PORT

runtime_sc:                                   # ← Decision 6 (declarative, no hook)
  qdrant:
    container:
      scale: 1
      environment:
        QDRANT_ENDPOINT: http://qdrant:6333
      deploy: {}
      extra_hosts: []
    localhost:
      scale: 0
      environment:
        QDRANT_ENDPOINT: http://host.docker.internal:${QDRANT_LOCALHOST_PORT:-6333}
      deploy: {}
      extra_hosts: [host.docker.internal:host-gateway]
    disabled:
      scale: 0
      environment:
        QDRANT_ENDPOINT: ''
      deploy: {}
      extra_hosts: []

data_flow:
  calls: []

# services/qdrant/compose.yml — Qdrant vector database
services:
  qdrant:
    image: ${QDRANT_IMAGE}
    container_name: ${PROJECT_NAME}-qdrant
    restart: unless-stopped
    deploy:
      replicas: ${QDRANT_SCALE:-0}
    ports:
      # Keep the HOST_BIND_IP prefix on every published port.
      - "${HOST_BIND_IP-127.0.0.1:}${QDRANT_PORT}:6333"
    volumes:
      - qdrant-data:/qdrant/storage
    healthcheck:
      # debian:13-slim base: no wget or curl, so probe the port with bash.
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 30s
    networks:
      - backend-network

volumes:
  qdrant-data:
    name: ${PROJECT_NAME}-qdrant-data
    driver: local

Add to the top-level include: block in the # Data tier section:

  # Data tier
  - services/supabase/compose.yml
  - services/redis/compose.yml
  - services/minio/compose.yml
  - services/neo4j/compose.yml
  - services/qdrant/compose.yml              # ← new

This is a mandatory registration step that's easy to forget. SourceOverrideManager.source_mapping is a hardcoded dict; the wizard's ServiceDiscovery.discover() filters every service through it. A service NOT in this mapping is silently dropped from the wizard — the user never sees a prompt for it.

For a single-container family (most services), add ONE entry:

self.source_mapping = {
    # … existing entries …
    'qdrant_source': 'QDRANT_SOURCE',          # ← new
}

For a multi-container family (head + worker, or app + init), the runtime_sc top-level key drives discovery. Map the "main" container's <key>_source to the family's actual env var. Example: Ray has ray-head and ray-worker in runtime_sc but one RAY_SOURCE env var. The mapping is 'ray_head_source': 'RAY_SOURCE'. The worker has no entry, so it is filtered out, as comfyui-init and hermes-init are.

The CLI flag binding in bootstrapper/start.py (@click.option('--qdrant-source', …) + the source_args dict) uses the family-level qdrant_source key — different from the discovery key for multi-container families. If your service is multi-container, you'll have TWO entries in source_mapping pointing to the SAME env var (one for CLI plumbing, one for discovery).

The pinning test bootstrapper/tests/test_wizard_app_discovery.py::test_source_mapping_includes_app_service_flags enforces this — add your service's CLI key to the assertion list so future regressions fail loudly.

Run these root-safe commands from the repository root:

# 1. Regenerate .env.example from manifests
uv run --project bootstrapper python -m services.env_assembler

# 2. Regenerate README.md TOPOLOGY block (auto-includes the new row)
uv run --project bootstrapper python -m tools.generate_readme_topology

# 3. Generate the per-service README and diagrams
#    (required when the new manifest owns a same-folder README)
PYTHONPATH=bootstrapper uv run --project bootstrapper python -m bootstrapper.docs.regen qdrant
# After this, services/qdrant/{README.md, architecture.svg, architecture.html} exist.

# 4. Update docs/diagrams/architecture.html and docs/diagrams/architecture.svg together
#    via the architecture-diagram skill if the full-stack topology changes.
#    Render the committed PNG, then refresh the generated surfaces.
uv run --project bootstrapper python -m scripts.docs.render_diagrams
make docs-build

# 5. Give every new `images:` entry a row in the supply-chain license inventory
#    (docs/reference/license-inventory.yaml) and check it
uv run --project bootstrapper python -m scripts.docs.license_inventory --check

# 6. Regenerate the canonical reference pages (env vars, sources, ports,
#    dependencies, manifest fields, license inventory, the validator catalog below)
uv run --project bootstrapper python -m scripts.docs.canonical_references

# 7. Lint manifests and all three documentation surfaces
uv run --project bootstrapper python -m tools.validate_fragments
make docs-check

When to re-run each step:

• env_assembler — after any change to a manifest's env: block, port allocation, or source variants.
• generate_readme_topology — after any change to a manifest's rows:, display_name, category, or alias.
• Top-level docs/diagrams/architecture.html and architecture.svg — hand-authored masters. Update both when a band-level change occurs, such as a new category or a new gateway. After either master changes, run python -m scripts.docs.render_diagrams as shown above before make docs-build. Routine data_flow.calls edits flow into per-service diagrams via bootstrapper.docs.regen.
• license_inventory — after adding an images: entry or moving any image pin or catalogue model source. The row records the image exactly as pinned and its license at the upstream revision of that version. It also records what the terms allow for hosted use, source integration and redistribution. Anything not yet adjudicated is marked unresolved with a named open item. See the supply-chain license inventory.
• canonical_references — after any manifest change. make docs-check fails on stale reference pages.
• validate_fragments — always, before the final make docs-check gate.
• docs.regen — required after creating a new service that owns a same-folder README, or after editing data_flow.calls on an existing service. Manifests whose docs: field points to an aggregate/doc-only README are exempt from same-folder generation. The drift gate in CI (bootstrapper.docs.regen --all --check) catches stale existing per-service READMEs/SVGs/HTMLs.

Prerequisite: Cairo. Diagram rendering uses cairosvg, which needs the native Cairo library: libcairo2 (Debian/Ubuntu) or brew install cairo (macOS). CI installs libcairo2.

Atlas hero art: regenerate with python bootstrapper/scripts/generate_logo.py (needs pip install pillow + chafa on PATH); commit the refreshed bootstrapper/ui/textual/assets/atlas_hero_*.json. Not gated in CI.

After adding a service, check these allowlists. A hard depends_on into a SOURCE-replaceable family fails CI unless ALLOWED_REPLACEABLE_DEPENDS_ON lists it. A default-on Kong route fails CI unless check-kong-routes.py lists it.

The script renders Compose against .env.example (or --env-file PATH), never your local .env, so it gives CI's answer everywhere. A hard depends_on fails unless it is listed below when it points into either:

• a container of another family whose manifest sources: offer localhost or disabled;
• an engine that such a source's endpoint names (for example speaches behind TTS_PROVIDER_SOURCE).

Edges inside one family, such as spark-worker → spark-master, are exempt.

• REQUIRED_DEPENDS_ON — reviewed (service, dependency) edges that must stay in the rendered Compose depends_on. The audit fails if one disappears. Add an edge here only to guard it. The script does not compare manifest depends_on with Compose.
• FORBIDDEN_OPTIONAL_DEPENDS_ON — edges that MUST NOT exist even though the rule above does not derive them, such as LightRAG's edges to stores it can replace in-process.
• ALLOWED_REPLACEABLE_DEPENDS_ON — reviewed exceptions to the derived rule: the dependent cannot work without that SOURCE-replaceable dependency and is only enabled alongside it (for example trino → iceberg-rest). Add an entry only with that justification.

The script runs the Kong route generator against .env.example defaults in a temporary directory. It checks the routes against the EXPECTED_HOST_ROUTES table at the top of the script. If your service publishes a *.localhost alias and its default <SVC>_SOURCE renders a route, add the host and its upstream URL to EXPECTED_HOST_ROUTES. Services that are off by default need no entry.

It also compares every default route by name, in both directions, against EXPECTED_ROUTES. It checks service, upstream URL, hosts, paths, strip_path, preserve_host, and service and route plugins. Global plugins are checked against EXPECTED_GLOBAL_PLUGINS. A path-only route, a dropped key-auth, or a flipped strip_path fails it. A default-on route you add or change needs its EXPECTED_ROUTES entry updated in the same change.

If your service ships a requirements.txt / pyproject.toml in a build/ or provider/ subdirectory, add the path to the directories: list of the pip ecosystem block. Note: all active manifests must be enumerated; omitted paths drop from scan coverage and silent vulnerabilities accumulate.

The .github/workflows/services-lint.yml workflow produces the four checks below. All four are required status checks in the live gitflow ruleset. Pull-request edits, including title edits, rerun the full workflow. A new push or edit cancels that pull request's superseded run. Runs on main and develop are never cancelled.

Manifest lint + unit tests aggregates these four jobs:

• Bootstrapper and Backend suites (with containers): validate_fragments, ShellCheck, and the pull-request title and changelog checks. It also runs the 7,000+ bootstrapper tests (with the container-backed backup/restore tests and the coverage floor) and the backend suite (services/backend/app/app/tests/).
• Bootstrapper suite without Docker (fast): the whole suite with no Docker daemon, so a failing unit test turns red first.
• Bootstrapper suite on Python 3.10: the full suite on the supported floor.
• MCP and asset API tests: those isolated suites.

Docs drift + audit scripts runs, in the audit-scripts job:

• regen --all --check and make docs-check;
• the audit scripts: check_doc_links (with #anchor validation), check-compose-source-deps, check-docs-drift, check-kong-routes, validate_research_schema, check-track-membership;
• notebook source hygiene;
• lock checks: Docling localhost provider, Local Deep Researcher, compiled runtimes, ComfyUI custom nodes, CI test locks;
• the ComfyUI custom-node overlay check and the container-image inventory and exception policy;
• a vulnerability audit of the compiled runtime locks.

Run this representative local subset before pushing (from the repository root unless a subshell changes directory). The authoritative command list is .github/workflows/services-lint.yml, which also runs ShellCheck, isolated MCP and asset-service suites, and the bootstrapper suite under Python 3.10:

(
  cd bootstrapper
  uv run pytest tests/ -q \
    --ignore=tests/test_fragment_equivalence.py \
    --ignore=tests/test_source_permutations.py \
    --cov=. --cov-config=pyproject.toml --cov-branch \
    --cov-report=term --cov-fail-under=69
)                                                                                  # job 1 bootstrapper tests
(
  cd services/backend/app/app
  BACKEND_TEST_VENV="${TMPDIR:-/tmp}/atlas-backend-ci-venv"
  uv venv --python 3.12 "$BACKEND_TEST_VENV"
  VIRTUAL_ENV="$BACKEND_TEST_VENV" uv pip install \
    -r requirements.txt -r requirements-dev.txt -c requirements-test-locked.txt
  "$BACKEND_TEST_VENV/bin/python" -m pytest tests/ -q -W error \
    --cov=. --cov-config=.coveragerc --cov-branch \
    --cov-report=term --cov-fail-under=79
)                                                                                  # job 1 backend tests
uv run --project bootstrapper python -m tools.validate_fragments                  # job 1 lint
docker compose --env-file .env.example -f docker-compose.yml config -q            # job 2 merge check
make docs-check                                                                    # job 3 three-surface contracts + strict build
uv run --project bootstrapper python scripts/check_doc_links.py                   # job 3 link check
PYTHONPATH=bootstrapper uv run --project bootstrapper python -m bootstrapper.docs.regen --all --check  # job 3 docs drift
uv run --project bootstrapper python scripts/check-docs-drift.py                  # job 3 docs structural audit
uv run --project bootstrapper python scripts/check-compose-source-deps.py         # job 3 deps audit
uv run --project bootstrapper python scripts/check-kong-routes.py                 # job 3 kong audit
uv run --project bootstrapper python scripts/validate_research_schema.py --all    # job 3 research schema
uv run --project bootstrapper python scripts/check-track-membership.py            # job 3 track coverage
uv run --project bootstrapper python -m scripts.bounded_subprocess \
  --label "Docling localhost lock check" \
  --cwd services/docling/provider/localhost -- uv lock --locked                  # job 3 docling lock
uv run --project bootstrapper python scripts/refresh-local-deep-researcher-lock.py --check  # job 3 Local Deep Researcher lock
uv run --project bootstrapper python -m scripts.check_runtime_locks               # job 3 compiled runtime locks
uv run --project bootstrapper python -m scripts.bounded_subprocess \
  --label "pip-audit tool installation" -- \
  uv tool install pip-audit==2.10.0                                                # job 3 pinned vulnerability-audit tool
uv run --project bootstrapper python -m scripts.audit_runtime_locks               # job 3 runtime vulnerability audit
# Final-image scan (not required): docker buildx build over every local Compose
# and init build context. The full, current list lives in that job's build loop
# in .github/workflows/services-lint.yml — treat that workflow file as the
# source of truth, not this doc. Build contexts validated
# in CI include services/asset-worker/app, services/asset-baker/app,
# services/backend/app, and others — representative example:
docker buildx build --load -f services/backend/app/Dockerfile services/backend/app/

A --<svc>-models-style flag (anything persisted via the wizard's user-model-selection path) needs FOUR coordinated edits, not two:

1. the @click.option declaration in start.py,
2. the source_mapping / collector-dict entry,
3. the user_model_selections[KEY] = kwarg assignment in main(),
4. the wizard bucket. _selections_to_args in ui/textual/integration.py returns it. The "Apply user model selections" lambda in ui/textual/screens/wizard_screen.py reads it with .get(...). A new bucket key needs both edits. Missing seam 4 silently drops the flag's value. AST-based seam-parity tests (tests/test_user_model_selections_seam_parity.py, tests/test_wizard_app_discovery.py) guard all four — extend them when adding a picker.

.env *_IMAGE values are refreshed from the manifests at every start (_refresh_image_pins_from_manifests()). To pin a different image, export the var in the shell before ./start.sh.

depends_on.required doubles as a runtime-boot gate and the wizard's display-order backbone — see Decision 5 for the full footgun and convention. In short:

• Don't drop a cross-category edge only because it is not a runtime dependency; it may pin wizard display order.
• Don't list a virtual aggregate (globals, cloud-providers, tts-provider) as a boot dependency; it has no container. A commented display-order pin is allowed.

<SVC>_LOCALHOST_PORT is the single source of truth for localhost URLs — see Decision 3 for the full rule. The gotcha: the in-container consumer, the Kong route generator and the wizard's inline input must read the same PORT var. Otherwise they silently disagree about where the localhost upstream lives.

• Kong routes fronting browser SPAs need preserve_host: True. Without it the SPA emits unreachable redirect URLs containing the internal Docker hostname.

• Init jobs never install OS packages at container startup. Bake tools into a service-owned build context (for example init/Dockerfile). Pin the multi-architecture base and every Alpine package version, and give the image a project-local name. Runtime apk add makes startup depend on mutable repositories and can silently change a deployed revision.
  • Application add-ons are the exception. n8n-init installs community nodes with npm ci from a committed lockfile. ComfyUI provisioning runs pip install for custom-node requirements.
  • A Debian image takes security fixes through a reviewed pinned-base refresh.
  • If the newest base still has fixable findings, a build-time apt-get upgrade is accepted, with the rationale written next to it.
  • The non-required Final-image scan job rescans each refreshed image.
• TTS/STT engine in-container ports are NOT all 8000. Parakeet, Speaches, Docling listen on 8000; Chatterbox listens on 4123. Don't assume.

• .env.example is byte-equivalence-tested. After any manifest change affecting env vars or port allocation, regen .env.example or test_env_example_consistency will fail CI.
• test_fragment_equivalence is sensitive to Compose-version defaults. Don't regenerate the golden baseline reflexively when this test fails. Extend _strip_volatile_defaults in the test fixture instead.

• A new service in a full category block trips the category-overflow lint. The infra block (10 slots) is full. A new infra *_PORT needs a new block or a different category. data has 30 slots (Supabase alone uses 8) and media 20. Check current utilization before assuming there's room: PYTHONPATH=bootstrapper python -c "from services.topology import get_topology; print(get_topology().port_defaults)".
• Renaming a row.display_name breaks tests that hardcode it. test_wizard_app_discovery.py has an EXPECTED_DISCOVERED frozenset; update it when renaming.

A service missing from SourceOverrideManager.source_mapping is silently dropped from the wizard — no error, the prompt just never appears. See §11.4 for the registration mechanics, including the two-entry pattern multi-container families need.

Each service folder can hold additional subdirectories beyond service.yml and compose.yml. Use these well-known names so navigation stays predictable across the stack:

Family folders. Some manifests own a family of related containers (e.g. supabase = 9 containers, n8n = main + worker + init). Each family folder gets a brief README.md listing the containers it ships and pointing at their definition in compose.yml. The fragment's services: keys are the authoritative container list; the manifest's containers: must match 1:1 (the manifest_validator enforces this — see "Validator rules" below).

• Env var default change → edit the manifest's env: block. Re-run the lint.
• Image version bump → see Bump a container image version below.
• New container in the family → add to containers: in the manifest AND to services: in the fragment.
• New source variant → add to sources.options[] in the manifest.

Short walk-throughs for the modifications you'll do most often:

• Add a new source variant to an existing service. Edit sources.options + runtime_sc.<key> to include the new source. Regen .env.example. The source-permutation matrix in CI will exercise every variant — make sure your new variant has a valid runtime_sc slice.
• Rename a service's display name (rows[].display_name). Update the row, then regen the README topology block. Search the test suite for the old name — bootstrapper/tests/test_wizard_app_discovery.py::EXPECTED_DISCOVERED is the most common dependency.
• Bump a container image version. Edit images[].default. The fragment uses ${X_IMAGE}, so it does not change. Also:
  • regenerate .env.example;
  • update the image line in the fragment baseline (§20);
  • update the image's row in docs/reference/license-inventory.yaml, then run make docs-check;
  • test the new image locally before you commit.
• Split a service family into multiple manifests. Non-trivial. The supabase manifest (9 containers in one family) is the reference pattern; consult it before splitting.

Service READMEs follow a numbered convention (## 1. Overview, ## 2. Access, …). The "Dependencies & Integrations" block sits at whatever section number N the README's structure places it — typically 5, but 7/9/12/14 in READMEs with extra pre-Deps content. The bootstrapper/docs/regen.py tool detects N and emits matching subsection numbering (### N.1 through ### N.6) inside the block.

Never link to a sub-section by number across services. "See section 5.4 in the backend README" breaks when a new pre-Deps section makes it 6.4. Always reference by heading text instead: "See Future — Missing pair integrations in the backend README."

This section is a quick field reference. For guidance on which fields apply when (and why), see Decisions 1–6 above.

name: myservice
label: "My service (human-readable)"
category: apps                          # infra | data | llm | media | agents | apps
docs: services/myservice/README.md        # safe repo-relative Markdown path
# docs_exception: "No guide is published because compatibility migrations preserve retired environment keys."
# Use only when neither docs nor a sibling README.md is available.
virtual: false                          # true for env-only manifests like cloud-providers

support:                                # required by tests; see §22
  tier: experimental                    # stable | experimental | community | unsupported
  evidence: "No qualification run cited yet."
  evidence_revision: v0.1.0             # release tag or full 40-char commit id

capabilities:                           # required by the schema; at least one entry
  - name: "Short capability label"
    status: supported                   # supported | partial | stubbed | not-supported
    verification: untested              # tested | documented | untested
    note: "One-line boundary or limitation."

containers:                             # may be [] when virtual: true
  - myservice

images:
  - var: MYSERVICE_IMAGE
    default: "myorg/myservice:1.0.0"
    container: myservice

sources:                                # OPTIONAL; omit for single-source services
  var: MYSERVICE_SOURCE
  default: container
  auto_prefer:                          # OPTIONAL; enables `MYSERVICE_SOURCE: auto`.
    - id: container                     #   Ordered; last entry must be unconditional.
  options:
    - id: container
      label: "Container"
    - id: disabled
      label: "Disabled"

# Per-source runtime data — what the bootstrapper actually consumes.
# The synthesizer (bootstrapper/services/sc_synthesizer.py) concatenates
# this block (plus all other manifests' runtime_sc blocks) into the dict
# that ConfigParser.load_yaml_config() returns.
runtime_sc:
  myservice:                             # the key matches a name service_config.py reads
    container:
      scale: 1
      environment:
        MYSERVICE_ENDPOINT: "http://myservice:8080"
      deploy: {}
      extra_hosts: []
    disabled:
      scale: 0
      environment:
        MYSERVICE_ENDPOINT: ""
      deploy: {}
      extra_hosts: []

env:
  - name: MYSERVICE_SOURCE
    default: container
  - name: MYSERVICE_PORT                # no default: the slot allocator assigns it
  - name: MYSERVICE_API_KEY
    default: ""
    secret: true                        # logged redacted; a default IS written to .env.example, so keep "" for real secrets
  - name: MYSERVICE_SCALE
    auto_managed: true                  # computed by service_config.py from source value
  - name: MYSERVICE_ENDPOINT
    auto_managed: true

depends_on:                             # logical deps (compose-level lives in compose.yml)
  required: []
  optional: []

exports:                                # documents the cross-service env-var contract
  - name: MYSERVICE_ENDPOINT
    consumers: [backend, n8n]

rows:
  - display_name: "My Service"
    source_var: MYSERVICE_SOURCE
    port_var: MYSERVICE_PORT
    scale_var: MYSERVICE_SCALE
    alias: myservice.localhost
    description: "Short description shown in the wizard subtitle."
    # OPTIONAL — inline numeric input mounted on the source prompt. Used by
    # Prometheus (TSDB retention days). The wizard
    # renders a SecondaryNumberInput widget next to the source picker so
    # the user picks source AND numeric refinement in one keystroke
    # sequence — no follow-up cascade step.
    secondary_number:
      env_var: MYSERVICE_KNOB
      label: "Knob value"
      default: "10"
      visible_when_source: ["container"]   # hide for `disabled`
      min: 1
      max: 100

Every service.yml first passes bootstrapper/schemas/service.schema.json. The table below is generated from the ordered registry that also executes the cross-manifest rules; do not edit it by hand.

bootstrapper/tests/test_fragment_equivalence.py asserts that docker compose -f docker-compose.yml config matches the golden baseline at bootstrapper/tests/fixtures/rendered_config_baseline.yml. If you change a fragment's compose shape, you'll either need to update the baseline (after confirming the change is intentional) or restore byte-equivalence.

The gate does not compare a raw docker compose config dump. It renders from .env.example plus a few test overrides and passes -p atlas. It also rewrites machine-specific absolute paths to the {REPO_ROOT} and {HOME} tokens the committed fixture stores. Redirecting docker compose config into the fixture does none of that. The result can never match the gate. That command also reads your live .env, so it writes real secrets into a tracked file.

Run these commands from the repository root.

# Inspect drift — the gate is the diff; it reports the first mismatching key path.
(cd bootstrapper && uv run pytest tests/test_fragment_equivalence.py -q)

# Refresh the baseline (only when the drift is intentional), reusing the gate's
# own helpers so the refreshed fixture is what the gate will compare against.
(cd bootstrapper && uv run python -c "
import subprocess, yaml
from tests.test_fragment_equivalence import _build_test_env, _normalize_paths, BASELINE, REPO_ROOT
rendered = subprocess.run(
    ['docker', 'compose', '--env-file', str(_build_test_env()), '-p', 'atlas',
     '-f', str(REPO_ROOT / 'docker-compose.yml'), 'config'],
    capture_output=True, text=True, check=True,
).stdout
BASELINE.write_text(yaml.safe_dump(_normalize_paths(yaml.safe_load(rendered)), sort_keys=True))
")

A full refresh re-sorts and reflows the whole (5,000+ line) fixture, which buries the change you actually made. For a one- or two-key change, edit the fixture surgically instead and let the gate confirm it.

Downstream repos that vendor Atlas as a git submodule can add services without editing the upstream tree. Put each one under services/_user/<name>/:

services/
├── _user/                    # gitignored upstream; tracked downstream
│   └── my-extra-service/
│       └── compose.yml       # required — auto-merged & launched (see below)
│                             # (a service.yml is optional; the core loader
│                             #  skips _-prefixed dirs, so it's only for your
│                             #  own tooling)
├── supabase/
├── ollama/
└── …

Auto-launch. At start, DockerManager._compose_file_args appends each services/_user/*/compose.yml after -f docker-compose.yml, so overlay services start and stop with the stack. With no overlay, the bootstrapper passes only -f docker-compose.yml, so the core stack and the byte-equivalence baseline do not change.

An overlay is a self-contained Compose fragment:

• It brings its own image, host ports and environment.
• It joins the shared network with networks: { backend-network: { name: ${PROJECT_NAME}-network, external: true } }.
• Prefix its published ports with ${HOST_BIND_IP-127.0.0.1:} to keep Atlas's loopback binding default.
• It is not wired into the wizard, the topology port allocator or the generated .env.example.
• bootstrapper.services.manifests.load_manifests skips _-prefixed folders, so a _user/<name>/service.yml is ignored. A wrapper can call load_manifests(services_dir / "_user") for the manifest model.

The upstream .gitignore excludes services/_user/, so the folder never leaks into a fork's pull requests. New consumers should prefer atlas.consumer.yml overlays (Reusing Atlas §6.1); _user/ stays for existing integrations. Walkthrough: Reusing Atlas §6.1.1.

A few fields on service.yml document ownership and capability contracts even though containers do not consume them directly:

• support: — the family's support tier and its evidence.

  • Required keys: tier (stable, experimental, community, unsupported), evidence and evidence_revision.
  • evidence is a one-line pointer to the qualification run, or a plain statement that none exists. evidence_revision is the release tag or full commit. Optional keys: owner and one-line limitations.
  • A selectable family is not thereby validated: every family starts experimental. stable needs cited cold-start, workflow and upgrade evidence from a release tag. The validator rejects a bare commit (support_stable_without_release_evidence), and a test checks that the tag is in the release record.
  • The tier appears in the generated catalog's Support column and at the top of each README's Capabilities & limitations section. Every non-stable family also gets a [support: …] badge on its wizard prompt.

• images[].notes — free-form note on what the image is used for. Not read by any Python code.

• docs: — repository-relative pointer to existing Markdown. The validator rejects missing, escaping, non-Markdown, directory, and symlink targets.

• docs_exception: — a last-resort reason when a manifest has no docs and no sibling README.md. Write one printable line with an explicit because clause followed by at least four substantive rationale words, including at least three distinct terms. The validator:

  • applies NFKC and case-folding, and rejects every Unicode category-C character (controls and zero-width formatting);
  • removes generic boilerplate and link-like text before locating and scoring the rationale. Link-like text is Markdown links/labels, angle autolinks, RFC URI schemes, email-like tokens, validated IP links and dotted host candidates;
  • treats as email-like an atom or quoted local part (spaces allowed) followed by a Unicode dotted host or bracketed domain literal. This grammar is deliberately not a full RFC email parser;
  • for hosts: Host candidates are IDNA-normalized. A valid dotted host with a port/path/query/fragment is stripped regardless of case, except invalid all-numeric candidates. Bare lowercase hosts with an alphabetic 2–63 character final label (or a punycode label) are also stripped. So an ambiguous lowercase identifier such as module.foo is removed;
  • keeps as technical context: bare uppercase module.Class, digit-suffixed identifiers and versions, single-label service DNS, numeric host-port pairs. It also keeps bare .local / .internal / .localhost names, including www-prefixed service DNS. Otherwise, suffix-bearing reserved DNS is stripped;
  • strips Validated IPv4 and bracketed IPv6 only with a port/path/query/fragment; bare IP mentions remain. Bracketed IPv6 may carry an optional non-empty zone identifier in raw %eth0 or RFC 6874 %25eth0 form. The identifier is bounded to [A-Za-z0-9._~-]+ and checked by the IP address parser;
  • strips, with a bounded fallback, malformed bracketed colon/address-like tokens that carry a port/path/query/fragment, so suffix prose cannot be scored. It does not assert that those fallbacks are valid addresses;
  • accepts a tracking link after a real reason, but not as the reason, and rejects an exception on a documented service.

  This is a bounded authoring gate, not proof that a rationale is semantically correct; reviewers still assess the concrete claim.

• exports[] — declares the env-var contract this service offers to other services. The cross-manifest validator (bootstrapper/services/manifest_validator.py) checks closure (every consumer name resolves) but does NOT check that the exported value is actually produced at runtime.

Every manifest, including virtual: true manifests, must resolve to safe documentation or carry a validated exception. Doc-only folders have no manifest and remain outside this rule.