LiteLLM is the always-on OpenAI-compatible front door for every LLM provider in the stack. The default consumers (Backend, Open WebUI, n8n, JupyterHub, Local Deep Researcher, OpenClaw, Weaviate vectorization) use one URL and one API key: LITELLM_BASE_URL and LITELLM_API_KEY. LiteLLM routes each request to the upstream for its model name. Services with a documented native-provider override, such as LightRAG role bindings, can bypass this path.

To open the dashboard or call the API, see §5 Access. To check a running gateway, see §10 Smoke tests.

When Hermes Agent is enabled, services/litellm/init/scripts/init.py appends a hermes-agent row to model_list, with api_base ${HERMES_ENDPOINT}/v1 and the os.environ/HERMES_API_KEY bearer token. The row does not come from the YAML model catalogs, because Hermes is a service, not a model provider. Open WebUI, n8n, backend, jupyterhub and openclaw see hermes-agent in their model lists with no per-consumer wiring.

• Image: ghcr.io/berriai/litellm:v1.83.14-stable.patch.2 (override via LITELLM_IMAGE). Pinned to a vX.Y.Z-stable tag; LiteLLM's prod docs explicitly warn against main-latest / main-stable.
• Internal port: 4000. Host port: LITELLM_PORT (default 63040). See .env.example for the current default.
• Internal port 4000 is also used by supabase-realtime; not a collision (different container hostnames).

Consumers ──► litellm:4000 ──► Ollama / vLLM Metal / Cloud providers
                  │
                  ├─► supabase-db:5432/litellm   (DATABASE_URL)
                  └─► redis:6379                 (REDIS_HOST/PORT/PASSWORD)

• Ollama upstream is a single-select choice (LLM_PROVIDER_SOURCE):
  • ollama-container-cpu / ollama-container-gpu — Docker Ollama upstream
  • ollama-localhost — Ollama running on the host machine
  • none — no Ollama upstream
• vLLM Metal is an independent managed-host choice:
  • VLLM_METAL_SOURCE=managed-localhost — native Apple-silicon vLLM upstream
  • VLLM_METAL_SOURCE=disabled — default
• Cloud providers are independent toggles (each enabled / disabled):
  • CLOUD_OPENAI_SOURCE (requires OPENAI_API_KEY)
  • CLOUD_ANTHROPIC_SOURCE (requires ANTHROPIC_API_KEY)
  • CLOUD_OPENROUTER_SOURCE (requires OPENROUTER_API_KEY)
• Bootstrapper refuses to start only when LLM_PROVIDER_SOURCE=none, vLLM Metal is disabled, AND every cloud source is disabled.

• Database: dedicated litellm database on the existing Supabase Postgres server. Created on first start by the litellm-init container (services/litellm/init/scripts/init.py). Same host (supabase-db) with a dedicated role (LITELLM_DB_USER / LITELLM_DB_PASSWORD) — separate logical database, so LiteLLM's LiteLLM_* Prisma-managed tables stay isolated from Supabase's schema. Override the database name via LITELLM_DB_NAME.
• Redis: response cache and rate-limit state, on the stack's Redis at redis:6379 with REDIS_PASSWORD.
  • The response cache serves embeddings, reranks and transcriptions for LITELLM_CACHE_TTL, because a fixed model returns the same result for the same input.
  • Chat, text-completion and Responses API calls are cached only when LITELLM_CACHE_CHAT=true. Then an identical request to hermes-agent or lightrag returns the cached reply for the TTL. It does not run the agent or see newly ingested documents. Send "cache": {"no-cache": true} in the request body for those calls.
• Master key: LITELLM_MASTER_KEY (must start with sk-) is auto-generated by the bootstrapper on first start and written into .env. Never overwritten on subsequent runs — virtual keys and spend history persist.

In the URLs and commands on this page, replace each ${VAR} with the value of that variable in .env.

The dashboard at /ui/ needs a login. Two env vars set the credentials on the LiteLLM container:

To recover the password, run from the repository root with the bootstrapper dependencies available through uv. The parser reads .env by default; use ATLAS_ENV_FILE for a custom active file. It prints the literal value and stops if the key is missing or blank:

PYTHONPATH=bootstrapper uv run --project bootstrapper python - <<'PYKEY'
import sys
from core.config_parser import ConfigParser

parser = ConfigParser(root_dir=".")
if not parser.env_file_path.is_file():
    sys.exit("The active Atlas env file does not exist.")
env = parser.parse_env_file()
key = env.get("LITELLM_MASTER_KEY", "")
if not key:
    sys.exit("Set a nonempty LITELLM_MASTER_KEY in the active Atlas env file.")
print(key)
PYKEY

Compose maps LITELLM_UI_USERNAME → UI_USERNAME and LITELLM_MASTER_KEY → UI_PASSWORD. LiteLLM requires both.

The Kong route at litellm.localhost is always on; there is no toggle. bootstrapper/utils/kong_config_generator.py::generate_litellm_service() generates it. It proxies the whole LiteLLM surface, not only /ui/, so every endpoint works on the direct port and on the alias. The route uses preserve_host: True. LiteLLM's SPA then builds login redirects to litellm.localhost:${KONG_HTTP_PORT}, which the browser can resolve, and not to litellm:4000.

The YAML model catalogs and the wizard selections in .env decide which provider models LiteLLM exposes.

services/ollama/models.yaml  ─┐
services/litellm/models.yaml  ├──► model_resolver.active_models(env) ──► litellm-init renders /app/config.yaml ──► litellm reads model_list
wizard selections in .env    ─┘

Whenever litellm-init starts, it calls model_resolver.active_models(). It reads the YAML catalogs and the wizard's env vars: LLM_PROVIDER_SOURCE, OLLAMA_USER_MODELS, OLLAMA_CUSTOM_MODELS, LITELLM_*_ENABLED, cloud *_API_KEY and *_USER_MODELS. It then renders volumes/litellm/config.yaml with the per-provider routing rules below. Config rendering uses no database query.

• For ollama-localhost with OLLAMA_AUTO_IMPORT_LOCAL_MODELS=true, litellm-init also reads the upstream /api/tags and adds host-pulled models to the active set. If the host daemon does not answer within 10 s, the init log prints auto-import: failed to fetch /api/tags. Host-pulled tags are then not registered for this run.
• Before docker compose up, the bootstrapper writes only a stub, for the bind mount. services/litellm/init/scripts/init.py fills in the real model_list before the LiteLLM proxy starts.

Change exposed models through the wizard (./start.sh), or edit their selection variables in .env before running it. For a model outside the curated catalog, set OLLAMA_CUSTOM_MODELS or add a YAML entry. For an embedding entry, declare its output dimension with dim:. The wizard lists models with dim: 768 first, and the chosen model's dim sets LANGMEM_EMBEDDING_DIM (see Ollama §5).

At selection and warm repair, memory dimensions use the first active alias from catalog and explicit model selections. The order is Ollama, OpenAI, Anthropic, then OpenRouter. A custom winner without dimension metadata requires an explicit LANGMEM_EMBEDDING_DIM. Init’s host auto-import happens later.

volumes/litellm/config.yaml is rebuilt on every run, so direct edits there are overwritten. Per-provider routing rules baked into litellm-init:

• Ollama rows: model: ollama/<name>, api_base: $LITELLM_OLLAMA_UPSTREAM. render_model_list registers two aliases per Ollama model: ollama/<name> and the bare <name>. Chat models use the ollama_chat/ adapter and embeddings use ollama/ (§7).
• OpenAI rows: model: openai/<name>, api_key: $OPENAI_API_KEY.
• Anthropic rows: model: anthropic/<name>, api_key: $ANTHROPIC_API_KEY.
• OpenRouter rows: model: openrouter/<name>, api_key: $OPENROUTER_API_KEY.

Each cloud row gets its provider prefix once. A name that already has it (the catalog's openrouter/... ids, a wizard-saved row) is kept. A bare vendor/model OpenRouter id from a flag or manifest therefore still routes through OpenRouter. Clients always use the configured name as model_name.

The wizard's multiselect choices persist as comma-separated lists in .env so they survive across runs:

To add a model outside the catalogs without the model picker, set *_USER_MODELS or OLLAMA_CUSTOM_MODELS in .env. Then run ./start.sh --no-tui and select the intended track when prompted. Atlas recreates init and the gateway so the generated model list takes effect. Plain docker compose up can leave an unchanged gateway running with its previous model list.

Note: model_resolver activates a provider only when it is enabled and keyed. Enabled means CLOUD_<PROVIDER>_SOURCE=enabled; the bootstrapper derives LITELLM_<PROVIDER>_ENABLED from it at every start. Keyed means <PROVIDER>_API_KEY is set. To stop routing to a provider, set CLOUD_<PROVIDER>_SOURCE=disabled. Do not edit LITELLM_<PROVIDER>_ENABLED; the next start overwrites it.

Default models follow the routed set. Before each launch, the bootstrapper checks defaults against this launch's registered models. The defaults are LITELLM_DEFAULT_MODEL, LITELLM_VISION_MODEL and the embedding pair (LITELLM_EMBEDDING_MODEL, LANGMEM_EMBEDDING_MODEL).

If a default is known to be unrouted, Atlas selects model_resolver's best active replacement. It prints Default model KEY: old -> new (reason). The embedding pair moves together. Known unrouted cases include:

• an ollama/* model while Ollama is not the engine;
• an Ollama tag that is not selected;
• a model of a disabled cloud provider;
• a cloud model that is no longer selected.

It never replaces a routed default. It also leaves a consumer-declared model, the managed vLLM Metal model, and an Ollama tag under ollama-localhost with host auto-import on. A model of an enabled provider whose API key is missing is kept too; fix the key. A stale default with no replacement is a warning.

litellm-init renders volumes/litellm/config.yaml at startup. Two things are bind-mounted into the container, deliberately kept in separate dirs:

The mounts are separate because catalogs cannot be nested inside the read-only /catalog mount, and /catalog must be on sys.path for litellm-init's loose sibling imports. Read init.py::_catalog_dir() before you change this wiring. A new resolver module that litellm-init loads must use try: from utils import X / except ImportError: import X. It then works in the bootstrapper venv and in the container. Coverage: bootstrapper/tests/test_litellm_init_loose_imports.py and test_compose_nested_mounts.py.

A downstream integration can expose its own OpenAI-compatible routes, usually served by a backend plugin, as LiteLLM models. Declare a litellm_models block in atlas.consumer.yml; Consumer Manifest Reference §9 has the merge, validation and secret-handling contract.

Atlas merges these rows into model_list at every start; it does not call the LiteLLM admin API. The consumer's aliases appear in /v1/models with no registration script. Removing the manifest removes only that consumer's rows on the next start. Coverage: bootstrapper/tests/test_consumer_litellm_models.py, test_litellm_init_render.py::TestConsumerModelMerge, test_consumer_doctor.py.

LiteLLM ships two Ollama integrations and they are not interchangeable:

Embedding endpoint source: the pinned LiteLLM adapter.

services/litellm/init/scripts/init.py::render_model_list uses the versioned catalog metadata below. Curated kind: embedding entries get ollama/; curated kind: chat entries get their declared adapter. A custom or live-discovered model without metadata uses the legacy name heuristic and logs a startup warning. Existing operator-defined models therefore keep working.

Catalog rows may declare metadata_version: 1. Its provider-neutral fields are kind (chat/embedding), adapter, capabilities, request_defaults (for example think: false), recommended_roles (extract, keyword, query, judge, embedding, vision) and an embedding's dim. The field schema is in the comments next to the loader in llm_catalog.py.

LiteLLM receives the standard model_info fields and a namespaced atlas_model_metadata block. Consumers can use it to select model roles without provider, model-family or hardware assumptions. A cloud model with no catalog row (for example one picked live from OpenAI) gets no top-level model_info.mode. LiteLLM's own model map then routes it, so a Responses-only model such as gpt-5-pro is not forced onto /v1/chat/completions.

These capabilities are declarations. ./start.sh models probe measures tool calling, JSON output, vision and embedding dimension through this gateway, on demand. It reports each declaration that does not hold and does not change model selection (see Operations §1.2).

Full metadata comes from the authenticated GET /v1/model/info. GET /v1/models is the compatibility listing and does not return the complete model_info payload:

curl -s http://localhost:${LITELLM_PORT}/v1/model/info \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}"

For extract, query or another role, clients can choose a model from this metadata with the following procedure. Then configure the selected model through LightRAG’s role bindings or the consuming client’s settings:

1. Keep rows with inferred: false, the required kind or capability, and the role in atlas_model_metadata.recommended_roles.
2. Apply an explicit operator preference when one is configured.
3. Otherwise use lexical (provider, catalog_name, model_name) order as the provider-neutral fallback.
4. Deduplicate Ollama's dual aliases by (provider, catalog_name) and retain the operator's preferred alias.

The order is deterministic and assumes no provider, hardware or model family.

Ollama's reasoning models (qwen3, gpt-oss, deepseek-r1, …) split their output into a thinking channel and a content channel. The response arrives empty when the max_tokens budget runs out inside the <think> block, or when a consumer such as Hermes Agent reads only content.

init.py applies think: false only when a model's catalog request_defaults declares it. The default qwen3.8:latest catalog row declares it, so its answer goes into content instead of reasoning. Other chat models do not inherit it. Chat models without metadata keep the old default and log a warning.

To get the thinking trace, override it per request:

curl -s -X POST http://localhost:${LITELLM_PORT}/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -d '{"model":"qwen3.8:latest","messages":[...],"think":true}'

think: false is not added to embedding entries, because their ollama/ adapter does not accept it.

• GET http://localhost:${LITELLM_PORT}/health/liveliness — fast liveness check (no auth).
• GET http://localhost:${LITELLM_PORT}/v1/models (with Authorization: Bearer $LITELLM_MASTER_KEY) — lists every model registered in model_list.

On the first database connection after a cold boot, the ${PROJECT_NAME}-supabase-db log shows a burst of relation "<…>" does not exist errors. These concern the configured LiteLLM database. This is an upstream LiteLLM race. Its 4 uvicorn workers each start Prisma and probe dashboard views while the views are being created. The errors clear within tens of milliseconds and do not recur.

Before both database checks, set LITELLM_DB_NAME to the value in the active env file. Unset or blank values use litellm. Use the raw name, not LITELLM_DB_NAME_URI. Confirm that all 8 dashboard views exist:

docker exec -e PGPASSWORD="$SUPABASE_DB_PASSWORD" ${PROJECT_NAME}-supabase-db \
  psql -U supabase_admin -d "${LITELLM_DB_NAME:-litellm}" -tAc "
    SELECT count(*) FROM pg_views WHERE schemaname='public' AND viewname IN
      ('LiteLLM_VerificationTokenView','MonthlyGlobalSpend','Last30dKeysBySpend',
       'Last30dModelsBySpend','MonthlyGlobalSpendPerKey','MonthlyGlobalSpendPerUserPerKey',
       'DailyTagSpend','Last30dTopEndUsersSpend')"
# Expect: 8

First confirm that the query reached the configured database and completed successfully. If views stay missing or startup errors persist, collect the init and gateway logs before escalation.

# Liveness
curl -s http://localhost:${LITELLM_PORT}/health/liveliness

# Chat completion (Ollama upstream)
curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -X POST http://localhost:${LITELLM_PORT}/v1/chat/completions \
  -d '{"model":"ollama/qwen3.8:latest","messages":[{"role":"user","content":"hi"}]}'

# Embeddings (Ollama upstream — Weaviate uses this path)
curl -s -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
  -X POST http://localhost:${LITELLM_PORT}/v1/embeddings \
  -d '{"model":"ollama/nomic-embed-text","input":"hi"}'

# Cache and spend audit
redis-cli -h localhost -p ${REDIS_PORT} -a "$REDIS_PASSWORD" --scan --pattern "litellm*"
psql -h localhost -p ${SUPABASE_DB_PORT} -U supabase_admin \
  -d "${LITELLM_DB_NAME:-litellm}" \
  -c "SELECT * FROM \"LiteLLM_SpendLogs\"
      ORDER BY \"startTime\" DESC LIMIT 5;"

• ollama-pull calls the Ollama upstream directly (/api/pull), because model pulls are not OpenAI-compatible. Compose sets its OLLAMA_HOST_URL from LITELLM_OLLAMA_UPSTREAM. It runs only for the ollama-container-* sources; for other sources OLLAMA_PULL_SCALE=0. For ollama-localhost, the bootstrapper pulls onto the host daemon instead (Ollama §5).
• Open WebUI's native Ollama UI features (model pulls in the UI) are disabled — ENABLE_OLLAMA_API: "false" is set in compose. To pull a model directly on an ollama-container-* source, run docker exec -it $PROJECT_NAME-ollama ollama pull <model>. On ollama-localhost, run ollama pull <model> on the host.

If LiteLLM stops being a fit (license change, security incident, project drift), Portkey AI Gateway (Apache-2.0) is the documented fallback. The default consumers read LITELLM_BASE_URL and LITELLM_API_KEY, so the swap happens there. Migrate any native-provider overrides separately.

When LIGHTRAG_SOURCE != disabled, litellm-init registers a lightrag model that proxies to LightRAG's Ollama-compatible API ({LIGHTRAG_ENDPOINT}/api). A prefix on the last user message selects the query mode: /local, /global, /hybrid, /naive, /mix or /bypass. Without a prefix the mode is mix.

curl -sX POST http://localhost:${LITELLM_PORT}/v1/chat/completions \
  -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
  -d '{"model":"lightrag","messages":[{"role":"user","content":"/hybrid What is RAG?"}]}'

Support tier: experimental — Capability contract declared; no cited cold-start, workflow, or upgrade qualification run yet (evidence at v0.1.0).

Open the full-size diagram for a full-screen view.

• litellm ↔ minio — Why: LiteLLM ships a first-class S3 logger that persists full request/response payloads. MinIO is in the stack but unused for LLM telemetry — wiring it gives offline replay, prompt-regression diffs, and audit trails without a new dependency. Mechanism: litellm_settings.success_callback: ["s3"] + s3_callback_params pointing at http://minio:9000; provision a litellm-logs bucket via minio-init. Effort: small. Confidence: high.
• litellm ↔ stt-provider — Why: LiteLLM exposes a unified /v1/audio/transcriptions endpoint, but consumers hit STT_ENDPOINT directly today, bypassing LiteLLM's auth/rate-limit/spend/Kong-alias affordances. Mechanism: add an audio_transcription row in model_list with model: openai/<name> + api_base: ${STT_ENDPOINT} (speaches is OpenAI-compatible; parakeet needs a thin shim). Effort: medium. Confidence: medium.
• litellm ↔ tts-provider — Why: same argument as STT — LiteLLM has /v1/audio/speech routing; consumers hit TTS engines directly. Mechanism: TTS row in model_list with model: openai/<voice> + api_base: ${TTS_ENDPOINT} for speaches; chatterbox (port 4123) is also OpenAI-compatible on /v1/audio/speech. Effort: medium. Confidence: medium.
• litellm ↔ searxng — Why: LiteLLM's MCP servers feature lets a tool be advertised to every chat-completions client. Wiring searxng as a built-in search_web tool gives open-webui/n8n/hermes/jupyterhub web search for free. Mechanism: define a LiteLLM MCP server in litellm_settings.mcp_servers calling http://searxng:8080/search?format=json. Effort: medium. Confidence: medium.

No high-confidence opportunities identified.

• Guardrails — Why pursue: presidio PII redaction, lakera prompt-injection scanning, hide-secrets — all configurable per virtual key. Stack handles user data but has zero LLM-side PII controls today. Effort: medium.
• Virtual keys + team budgets — Why pursue: TrueForge already uses a dedicated virtual key, with a master-key fallback on mint failure. Other stock consumers use the master key by default. Atlas provisions no per-service spend caps or team budgets. Extending virtual keys to n8n / jupyterhub / open-webui would support separate budgets and credential revocation. Effort: small.
• Semantic caching + per-key cache controls — Why pursue: basic Redis response caching is already enabled stack-wide; LiteLLM's embedding-similarity semantic cache and per-virtual-key TTL/namespace controls remain unused. Effort: small.
• /v1/audio/transcriptions + /v1/audio/speech routing — Why pursue: see pair-integrations above. Effort: medium.
• Fallback model chains — Why pursue: declare fallbacks: [{"gpt-5": ["claude-opus", "ollama/qwen3.8"]}] so a cloud outage degrades gracefully to local Ollama. Effort: small.