Model Hotel has two related but distinct routing concepts: transparent failover and hotel routing. Together they ensure requests succeed even when individual providers are unreliable.

1. Architecture Overview
2. Database Schema
3. Failover Groups
4. Hotel Routing
5. Provider Selection Algorithm
6. Transparent Failover
7. Circuit Breaker
8. API Reference
9. Request Flow Sequence
10. Cache Behavior
11. Combined Behavior
12. Troubleshooting
13. Performance Considerations
14. Security Notes
15. Related Documentation

CREATE TABLE model_failover_groups (
    id              UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    display_model   TEXT NOT NULL UNIQUE,        -- Base model name (e.g. "glm-4.6")
    display_name    TEXT,                        -- Human-readable label (optional)
    description     TEXT DEFAULT '',             -- Human-readable description
    priority_order  JSONB NOT NULL,              -- [uuid1, uuid2, uuid3] - ordered list of model UUIDs
    entry_enabled   JSONB DEFAULT '{}',          -- {uuid1: true, uuid2: false} - per-entry toggle
    group_enabled   BOOLEAN DEFAULT true,        -- Master toggle for entire group
    auto_created    BOOLEAN DEFAULT false,       -- true if auto-generated during discovery sync
    auto_disabled_at  TIMESTAMPTZ,               -- set when discovery/sync disabled the group (below 2 routable members)
    floor_disabled_at TIMESTAMPTZ,               -- set when the dashboard's member-toggle cascade disabled it
    created_at      TIMESTAMPTZ DEFAULT now(),
    updated_at      TIMESTAMPTZ DEFAULT now()
);

Field Details:

Indexes:

• Unique index on display_model (enforces one group per base model name)
• No additional indexes needed (lookups by display_model or id)

-- models table (references providers)
CREATE TABLE models (
    id            UUID PRIMARY KEY,
    provider_id   UUID REFERENCES providers(id) ON DELETE CASCADE,
    model_id      TEXT NOT NULL,              -- Full model ID (e.g. "z-ai/glm-4.6")
    enabled       BOOLEAN DEFAULT true,
    UNIQUE(provider_id, model_id)
);

-- providers table
CREATE TABLE providers (
    id            UUID PRIMARY KEY,
    name          TEXT NOT NULL,
    base_url      TEXT NOT NULL,
    encrypted_key BYTEA,                      -- AES-256-GCM encrypted API key (NULL for a keyless provider)
    key_nonce     BYTEA,                      -- random AES-GCM nonce (NULL for a keyless provider)
    key_salt      BYTEA,                      -- per-key Argon2id salt for deriving the encryption key
    enabled       BOOLEAN DEFAULT true
);

A failover group represents one logical model backed by multiple providers. When you request hotel/glm-4.6, the proxy looks up the failover group with display_model = 'glm-4.6' and tries providers in the configured priority order.

The Failover Groups page includes a Manage Providers button (next to "New Group") that opens a modal listing all providers. Disabling a provider in this modal disables all model entries belonging to that provider across all failover groups (including custom groups). The same rule the backend applies is applied here as the toggle is sent: a group left with fewer than two routable entries is disabled with it, and a group that regains two is re-enabled. A toast notification reports how many groups were affected.

Auto-Generation (Discovery Sync):

Failover groups are automatically created during model discovery sync (SyncAllModels):

1. Query all enabled models from enabled providers
2. Strip org prefixes (openai/, deepseek/, meta-llama/, etc.) to get base name
3. Group providers that share the same base name
4. Require 2+ enabled model rows - the count is of enabled model rows (on enabled providers) sharing the base name, so two ids on one provider count as two. An auto-created group whose base name drops below two such rows is deleted, not disabled
5. Set auto_created = true
6. Leave custom groups alone - a custom group whose name equals a base name (created while only one provider served the model, renamed to it, or imported from the fleet primary) is never adopted or deleted by the auto rules; its members and toggles stay yours. The custom-group rules still apply: stale entries are pruned and it is switched off below two routable members

In the dashboard an auto-created group carries an "auto" badge. Each entry shows its effective state: an entry whose model or provider is disabled is greyed out with a badge, since the router skips it regardless of the entry toggle.

Groups are modality-agnostic: an embedding, rerank, image or audio model forms a group by the same rule and is reached through the same hotel/<name> on its own endpoint (hotel/rerank-v3.5 on /v1/rerank). A model only one provider lists gets no group, whatever its modality; that request is 404 model not found: hotel/<name> until a second provider lists the same id.

// internal/failover/sync.go:SyncAllModels
func (r *Repository) SyncAllModels(ctx context.Context) (*SyncResult, error) {
    // 1. Query enabled models of enabled providers
    rows, _ := r.pool.Query(ctx, `
        SELECT m.id, m.model_id, m.provider_id, p.name
        FROM models m
        JOIN providers p ON m.provider_id = p.id
        WHERE m.enabled = true AND p.enabled = true
        ORDER BY m.model_id, p.created_at ASC
    `)

    // 2. Group by base name (the segment after the last "/", lowercased)
    baseToModels := make(map[string][]modelInfo)
    for rows.Next() {
        base := normalizeBaseModel(modelID)  // "z-ai/glm-4.6" → "glm-4.6"
        baseToModels[base] = append(baseToModels[base], ...)
    }

    // 3. Two or more providers: create or refresh the auto group.
    //    Fewer: delete the auto group if one exists.
    for base, models := range baseToModels {
        if len(models) <= 1 {
            r.deleteUndersizedAutoGroup(ctx, base, len(models), providerNames, result)
            continue
        }
        r.upsertAutoGroup(ctx, base, currentIDs)
    }
}

After that loop the sync does three more passes: any auto group whose base name no longer appears at all is deleted, stale entries (members whose model row is gone) are pruned from every group, and custom groups left with fewer than two routable members are auto-disabled. A member counts as routable when its model and its provider are both enabled; the per-entry toggle is ignored there, because that is a reversible choice the router already honors. Custom groups are disabled rather than deleted so hand-built membership survives until the missing member returns.

Manual Creation (API):

POST /api/failover-groups
Content-Type: application/json
Authorization: Bearer $ADMIN_TOKEN

{
  "display_model": "glm-4.6",
  "display_name": "GLM 4.6",
  "description": "Multi-provider failover for GLM 4.6",
  "entry_ids": [
    "uuid-of-openai-gpt4o-model",
    "uuid-of-deepseek-gpt4o-model",
    "uuid-of-zai-gpt4o-model"
  ]
}

Validation Rules:

• Minimum 2 entries required for creation
• display_model must be unique (conflict if exists)
• display_name and description are optional

Priority Ordering:

Providers are tried sequentially in priority_order array order. Reorder via API or UI drag-and-drop:

PUT /api/failover-groups/{group_id}
Content-Type: application/json

{
  "priority_order": [
    "uuid-3",  // Now first (highest priority)
    "uuid-1",
    "uuid-2"   // Now last
  ]
}

Per-Entry Enable/Disable:

Individual entries can be disabled without removing them from the group:

{
  "entry_enabled": {
    "uuid-1": true,
    "uuid-2": false,  // Skipped during routing
    "uuid-3": true
  }
}

Disabled entries:

• Shown with a dashed outline in the UI
• Skipped during candidate selection
• Preserved in priority order for re-enablement

Effective Entry State:

The per-entry toggle expresses intent; whether the router can actually use an entry also depends on the underlying model and provider being enabled. Each entry in API responses therefore carries model_enabled and provider_enabled alongside the toggle's enabled, and the UI greys out entries whose model or provider is disabled and badges them ("Model disabled" / "Provider disabled") even when the entry toggle is on. The toggle stays functional so intent is preserved for when the model comes back. In the group editor, entries that are no longer available are suffixed "(unavailable)" so they stay distinguishable from same-named available candidates.

Self-Heal on Discovery:

When a discovery scan disables a model (it left the provider's listing - e.g. a rename), the scan re-syncs that model's failover group the same way a manual sync would: the stale entry is pruned, an auto-created group that drops below two enabled members is deleted, and a custom group that drops below two routable members is disabled. The changes are reported in the scan's discovery summary. See Model Discovery for the diff format.

Group-Level Toggle:

Master switch disables entire group:

{
  "group_enabled": false
}

When group_enabled = false:

• hotel/{display_model} requests return 404
• Group appears in UI but marked as disabled
• A custom group is switched off this way automatically once it has fewer than two routable members; an auto-created group in that position is deleted instead
• On an auto-created group the master toggle does not stick: every discovery scan or sync rewrites an auto group with group_enabled = true (its per-entry toggles and priority order are kept). To keep a shared model out of hotel/ routing, disable its entries or the models themselves instead

DELETE /api/failover-groups/{group_id}
Authorization: Bearer $ADMIN_TOKEN

Cascade Behavior:

• Deleting a failover group does NOT delete underlying models or providers
• Only the failover configuration is removed
• Direct provider routing (e.g. OpenRouter/z-ai/glm-4.6) continues to work

Hotel routing is explicit multi-provider routing via the hotel/ prefix. It gives you fine-grained control over which providers are used and in what order.

Prefix any model ID with hotel/:

curl -X POST http://localhost:8081/v1/chat/completions \
  -H "Authorization: Bearer $PROXY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "hotel/glm-4.6",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

This resolves to the failover group named glm-4.6 and tries providers in priority order.

The prefix itself is matched case-sensitively: hotel/ routes through a group, Hotel/ does not, and is treated as an ordinary provider-prefixed model name that will not resolve. What follows the prefix is lowercased before the lookup, so hotel/GLM-4.6 and hotel/glm-4.6 reach the same group.

Three conditions all answer 404, each with its own reason in the error body: no group exists for that name (model not found: hotel/<name>), the group's master toggle is off (failover group disabled), or the group has no entries at all (no entries in failover group). A group that exists and is enabled but whose every entry was filtered out answers with the no-available-provider error instead, which carries the reason each candidate was skipped.

The other form, <provider>/<model>, names one provider directly and carries the provider's name in the form routing uses: every space becomes a hyphen, so a provider named my provider is addressed as my-provider/<model>. Provider names are unique in that form, so an id names exactly one provider.

Exact Base Name Matching:

Hotel routing matches exact base names after stripping organization prefixes:

Base Name Extraction:

The code does not maintain a hardcoded prefix list. Instead, normalizeBaseModel() in internal/failover/sync.go strips everything before the last / in the model ID, then lowercases the result:

func normalizeBaseModel(modelID string) string {
    if idx := strings.LastIndex(modelID, "/"); idx >= 0 {
        return strings.ToLower(modelID[idx+1:])
    }
    return strings.ToLower(modelID)
}

This handles any org prefix depth (e.g. zai-org/glm-5.1, zai-org/anthracite-org/magnum-v4-72b) without needing to enumerate known providers. The SQL in SyncForModel uses SUBSTRING(m.model_id FROM '[^/]+$') to match on the same leaf segment.

Example: z-ai/glm-4.6, zai-org/glm-4.6, and glm-4.6 all map to the same failover group glm-4.6.

resolveHotelModel in internal/proxy/resolve.go runs in phases:

1. Group lookup (lookupFailoverGroup, cache first): a missing group answers model not found: hotel/<name>, a disabled one failover group disabled, an empty one no entries in failover group (all 404).
2. Entry filter (enabledEntryIDs): keeps the entries whose per-entry toggle is on, in priority_order.
3. Batch loads: one GetByIDs for the models, one for their providers.
4. Candidate build (buildFailoverCandidates): skips a missing or disabled model, a missing or disabled provider, and (when circuit_breaker_enabled) a candidate the breaker blocks, noting each breaker skip with when it can be retried; then decrypts the provider key (a keyless provider yields an empty key) and appends the candidate.

If every candidate failed key decryption the request errors with all N candidate(s) failed key decryption (wrong master key?); otherwise the ordered candidate list and the breaker-skip summary go to the failover loop.

Failover Groups page - showing groups with enabled status, model entries, and priority badges

Failover group entries with per-entry enable/disable toggles and priority ordering.

The provider selection algorithm builds an ordered list of candidates from a failover group:

Key Properties:

1. Priority order preserved: Candidates are always in priority_order sequence
2. Graceful degradation: Disabled/failed entries are skipped, not fatal
3. Circuit breaker integration: candidates whose own model circuit is open are filtered out before the failover loop, and so is every candidate of a provider the breaker has sidelined as a whole. That whole-provider judgement is derived: it is recomputed from the provider's model circuits on every read rather than stored, and it holds once enough of them are open at once (see Provider-wide verdict)
4. Batch efficiency: Single DB query for all models, single query for all providers

Transparent failover happens automatically when an upstream provider request fails with a retriable error. It is the proxy's mechanism to ensure requests succeed even when individual providers are flaky or misconfigured.

Failover applies to all /v1 endpoints, not just chat completions - the multimodal endpoints (embeddings, images, audio) share the same candidate-resolution and failover loop. For streaming or binary responses, failover is only possible before the first response byte has been forwarded to the client; once a stream is committed, the proxy stays with that provider.

The diagram below focuses on the streaming safeguards - the TTFT probe that gates commit, the stall watchdog that guards a committed stream, and the backoff-paced retry loop. The Architecture Overview diagram above shows the complementary resolution-to-selection pipeline.

Failover is sequential by default - providers are tried one at a time, in order (for streaming groups, opt-in Request Hedging races them instead):

1. Client requests a model (e.g. hotel/glm-4.6 or OpenRouter/z-ai/glm-4.6)
2. The proxy resolves a list of candidate providers
3. The first candidate is tried
4. If the upstream returns a retriable error, the proxy moves to the next candidate
5. This continues until the request succeeds or all candidates are exhausted
6. If all candidates fail, the answer follows the last failure:
   • the last candidate answered an HTTP error: that upstream status is passed through, with a gateway-written message naming the classified reason, the provider and the upstream status (the provider's own body stays in the request log; a payload-class refusal such as a 400 or 422 is forwarded as the provider wrote it)
   • a transport failure or a failed TTFT probe: 502
   • the failover or retry deadline expired: 504
   • every candidate was busy (a saturated 429, or its provider sat at its in-flight window): 429 with a Retry-After header (see "429s: saturated vs exhausted" below)
   • the client disconnected: 499

Failover triggers on:

Failover does not trigger on:

A 400 is the provider naming something in the request it will not accept, so the retry goes back to the same provider with that thing changed. Every rejection below is learned and cached per model, so later requests skip the round-trip.

• An unsupported parameter is dropped. For example an injected chat_template_args, or a thinking block a strict backend rejects.
• A renamed token budget is renamed, not dropped. When a model demands max_completion_tokens instead of max_tokens (OpenAI gpt-5 / o-series), the value is carried across so the response stays capped.
• A refused response format is rewritten. DeepSeek answers This response_format type is unavailable now to json_schema, which its API does not serve; the retry goes out in JSON mode (json_object) with the schema folded into the leading system message. DeepSeek and Z.AI Coding Plan providers, whose docs list JSON mode alone, get that rewrite on the first attempt. A GLM model on any other provider type keeps json_schema on the request and gets the schema in the prompt as well: it takes json_schema without error and answers in a shape of its own otherwise. The provider does not validate against the schema, so strict is best effort there.
• Tools plus reasoning move to /v1/responses. OpenAI's newest models reject that combination on Chat Completions entirely; the retry re-issues the request against the provider's /v1/responses (translated both ways), and later tools+reasoning requests route there preemptively.

See API Reference for the request shapes.

A 402 is the status-code sibling of an "insufficient balance" 429: the provider is saying a person has to pay before anything works again. It fails over like any retriable status, and the model's circuit opens on that one answer rather than serving out a threshold of failures, pinned for as long as pinning allows (the gateway's hint is 90 days, clamped to circuit_breaker_quota_pin_max). It is the same operator switch as an exhausted 429: with circuit_breaker_open_on_exhaustion off, a 402 is charged as an ordinary failure instead.

The charge lands on the model that drew it, not the whole account, because a 402 carries no body the classifier reads. At the default circuit_breaker_span_models of 2 the second model to take a 402 therefore sidelines the provider. That is the right default for a billing block, which is account-wide by nature, and the wrong one for a provider that answers 402 per request (OpenRouter does, when one request's maximum cost exceeds the balance), where two large requests can bench models that would have served.

Each failover attempt includes an exponential backoff delay to prevent overwhelming failing providers:

• Base delay: 100ms
• Maximum delay: 2s
• Jitter: [0, base) (adds 0 to base-1ms random offset, not symmetric)
• Before the first contact: no delay. The backoff counts providers actually contacted, so a candidate skipped without a request (its provider was at its in-flight window) does not advance it either
• Client disconnection during backoff: Returns HTTP 499

The delay doubles with each subsequent attempt (100ms → 200ms → 400ms → 800ms → 1.6s → capped at 2s), with jitter applied to prevent thundering herd issues.

// internal/proxy/proxy.go:failoverBackoff
func failoverBackoff(base, capacity time.Duration, attempt int) time.Duration {
    exp := min(time.Duration(float64(base)*math.Pow(2, float64(attempt-1))), capacity)
    jitter := time.Duration(rand.Int64N(int64(base)))
    return exp + jitter
}

By default, a 429 rate-limit response from one provider triggers failover to the next. This is controlled by the failover_on_rate_limit setting (default: true). If you prefer to return 429 errors directly to the client rather than retrying on another provider, set this to false.

A 429 makes one of two very different claims, and the proxy reads the response (body phrases, Retry-After, X-RateLimit-Reset* headers, and a wait the body itself states such as Google's RetryInfo detail or a "please retry in 2h35m" sentence) to tell them apart (rate_limit_classify_enabled, default true). A wait the provider states outranks the phrase table's default. Above the saturation ceiling (rate_limit_saturation_max_wait), a Retry-After or a wait stated in the body pins the model for that long, so a daily quota that names its reset is pinned for the reset and not for the until-paid ceiling. At or under the ceiling, a wait means saturation when no phrase matched; against a matched exhaustion phrase only a structured retry detail in the body (Google's RetryInfo) turns it into saturation, since a short Retry-After or an aggregator's free-text "try again in 30 seconds" can wrap an out-of-credit refusal that no retry fixes.

• Saturated ("busy"): a concurrency slot or a per-minute budget is full and a retry succeeds in seconds. The request still fails over, but the circuit breaker is not charged: a provider at capacity is healthy, and benching it wastes the very slots that are busy serving. When the last candidate is merely saturated, the proxy waits out the provider's Retry-After (capped by rate_limit_saturation_max_wait, default 60s) and retries that candidate once before giving up.
• Exhausted ("spent"): a usage window or balance is gone and retrying cannot succeed until it resets. One such response opens the model's circuit outright (circuit_breaker_open_on_exhaustion, default true) and pins its cooldown from the response's own claim; a quota-advisor reading always overrides that inferred pin (the status API reports which via pin_source).
• Unknown: anything unrecognizable keeps the old behavior and charges the breaker as an ordinary failure. As a fallback, an unknown 429 from a model that served a success inside rate_limit_recent_success_window (default 60s) is treated as saturated, so the mechanism works even for providers whose wording the phrase table has never seen.

Terminal responses are honest about the difference (failover_exhaustion_status_429, default true): when every candidate is busy, the client gets a 429 with a Retry-After instead of a 502, and when the group resolved to no candidates with at least one of them skipped by the breaker, the 429 names the earliest instant a circuit comes back, capped at 60 seconds so a long quota pin never tells a client to go away for hours (error_kind is provider_saturated, or provider_quota_exhausted when every skip is waiting out a quota pin). A group emptied with no breaker skip at all (every entry disabled or missing) answers 502 no available provider for hotel/<name>.

All of this only applies when the 429 was failover-eligible: with failover_on_rate_limit off, a 429 is forwarded to the client exactly as before.

Providers with no usage API. Ollama Cloud's account API names the plan and never the usage; a plain OpenAI-compatible endpoint exposes nothing at all. For those the only quota reading the gateway ever gets is the exhausted 429 itself, so the gateway keeps each provider's last one (the phrase the classifier matched, the model that drew it, and when) and the provider card shows it beside where a usage badge would be: no usage API · cap hit 14:51, with last cap message: "session usage limit" at 14:51 (from a 429 on gpt-oss:120b) on hover. An operator who saw a "90% used" email then knows that was the last real reading and that the cap has been hit since, without guessing which window. A spent balance or plan (an exhaustion a person fixes, entitled on the note) reads balance spent rather than cap hit. The ledger is written whether or not failover may act on the 429 (failover_on_rate_limit is a routing choice, and the 429 is forwarded untouched when it is off); only rate_limit_classify_enabled off leaves it blank, since then nothing is classified. It is in memory (GET /api/providers carries it as last_cap); a restart forgets it and the next exhausted 429 writes it again. A provider with a usage badge shows its real reading instead.

Transient 5xx on the last candidate. The last candidate gets the same one retry when it answers a transient 5xx (500, 502, 503 or 504; a 501 is a refusal a retry cannot change): a short jittered backoff (about 0.5 to 1s), then the same candidate once more, so a provider's one-shot "internal network failure, please try again later" is not the client's answer when there is nobody else to ask. This is what a request to a single provider (no hotel/ prefix) gets instead of failover. The retry appears in the attempt trail as a further attempt, the breaker is charged for each real 5xx, and the retry is skipped when the request's overall deadline has no room for it. server_error_retry_enabled (default true) switches it off, in which case the first answer is forwarded as before. Like failover itself, the retry re-sends the request, so a provider whose gateway timed out on a generation it had in fact finished may bill that generation twice; that at-least-once trade is the same one cross-provider failover already makes.

Transient network failures on any candidate. A transport error that is safe to repeat is retried against the same provider up to twice more before the loop moves on: any failure before the request reached the wire (DNS, dial, TLS handshake), and after it only a connection interruption (reset, broken pipe, unexpected EOF, an idle connection closed by the server). Timeouts and cancellations are never retried this way. The retries run inside the attempt's own timeout, and the breaker is charged at most once for the attempt, after they are spent, so a blip that heals on retry never counts against the provider.

Saturation happens because the gateway sends a fourth request to a provider that can take three, and almost no provider states its concurrency limit in a machine-readable way. So the gateway learns it the way TCP does (inflight_limiter_enabled, default true): every saturated 429 cuts the provider's allowance to one below what was in flight when it arrived, inflight_grow_after consecutive clean completions grow it back by one, and after inflight_forget_after without a cut the allowance returns to uncapped, so a provider that never saturates costs nothing. A remaining: 0 in the provider's own rate-limit headers caps the allowance immediately, without waiting for the 429.

The router respects the learned allowance the way it respects an open circuit: a candidate whose provider is at its window is skipped without a request and the next entry in priority takes it. When every live entry is at its window, the request waits for the first slot to free on any of them (bounded by rate_limit_saturation_max_wait and the overall deadline) and is sent there; that is the only place strict priority is not honored, and only because the preferred entry had no slot to honor it with. If nothing frees in time the client gets the honest 429 with a Retry-After.

The state is per member and in memory (runtime health, never synced): four members each learn their own allowance against a shared pool, and the sum converges because each member's cuts are driven by the 429s it draws itself. Per provider, not per model - slots are an account property, while the circuit breaker stays per model (full vs broken are different questions). Two gauges expose it: modelhotel_provider_inflight_limit (0 = uncapped) and modelhotel_provider_inflight.

An operator who knows the real number can set a hard ceiling per provider (max_in_flight in the provider's edit dialog, empty = none, synced with the fleet config); the learner still runs underneath it.

The proxy evaluates whether an upstream response should trigger failover based on the HTTP status code:

// internal/proxy/resolve.go:shouldFailover
func (h *Handler) shouldFailover(ctx context.Context, statusCode int) bool {
    if statusCode >= 500 {
        return true
    }
    switch statusCode {
    case 429:
        return h.settingsRepo.GetBool(ctx, "failover_on_rate_limit", true)
    case 401, 402, 403, 404, 499:
        return true // another candidate can serve every one of these
    }
    return false
}

Note: failover only proceeds if there are remaining candidates. If the last candidate returns a retriable error, its status is returned to the client with the gateway's classified message (see step 6 of How It Works).

The proxy uses the request_timeout setting (default: 1m) for all upstream requests:

• Non-streaming requests: Uses the setting value directly
• Streaming requests: Uses 10× the timeout value (10 minutes by default), to accommodate reasoning models that can take minutes before the first token. The TTFT probe still expects a frame carrying output within ttft_timeout, so a model that thinks for minutes without streaming its reasoning needs that setting raised too
• Long-running multimodal requests (image generation, audio): the same 10× value, since their legitimate latencies run just as long without carrying a stream flag

The value is read once before the failover loop, so every attempt in one request uses the same timeout even if the setting changes mid-request.

That is the per-attempt timeout. The request as a whole also carries an overall deadline of twice that value (2 minutes by default, 20 for streaming and long-running requests), counted from arrival: a second candidate gets its full time, and any later candidates share what is left. When it passes, the request ends with a 504.

This setting is configurable at runtime via the Settings UI or the admin API.

The request body is cached in the context at the start of the failover loop. This allows the body to be re-read for each retry without requiring the client to resend it. The cached body is used when building upstream requests for each failover attempt.

For streaming requests, the proxy reads ahead to confirm the first token arrives before committing the stream to the client. This prevents the client from receiving a broken or partial stream from a provider that responded 200 but fails to produce content.

• Timeout: Configurable via ttft_timeout setting (default: 1m0s). If the provider fails to produce a token within this timeout, the request fails over to the next provider. A token is a frame carrying model output (content, reasoning, a tool call, an Anthropic content block carrying text, thinking, tool input or a tool name); a role-only opener, a usage-only chunk, an Anthropic message_start or a keepalive comment does not satisfy the probe, so a provider that answers 200 and then pings for minutes without producing anything is failed over before any byte reaches the client. A stream that ends behind such frames is an empty answer: it is committed, forwarded with its own finish reason, and charged to the breaker once. The window also covers prompt processing on the native Anthropic and Anthropic-translated paths, whose message_start arrives before the prompt is read; raise ttft_timeout where very large prompts are routine.
• Measurement: The actual TTFT measured during the probe is stored in logEntry.ttftMs for request logging.
• Disable: Set ttft_timeout to 0s to disable the probe (immediate stream commit, backward-compatible behavior).

Sequential failover only reacts once a provider has actually failed, so a provider that is merely slow to its first token costs the client that whole wait. Request hedging (Hedge Slow Streams in the Settings UI, hedging_enabled, default false) races the candidates instead: the first candidate is sent immediately, and each Hedge Delay (hedge_delay, default 4s, UI slider 1 to 15 seconds) that passes with no first token launches the next candidate in parallel. A slot freed by an attempt that failed launches the next candidate at once rather than waiting out the delay.

The first attempt to confirm a first token (or to complete as an empty answer) wins. The orchestrator cancels every other attempt still in flight, stamps the winner's provider and model onto the request log, and streams the winner to the client. Nothing a loser produced ever reaches the client.

The losers are still paid for. Every attempt whose provider answered with a 2xx and that did not serve, whether it lost or was cut in flight, has had its prompt read and billed, so the request is charged for it: an estimate of the prompt from the request text (an abandoned stream never reports usage) goes to the key's token limits and counter, and the row's cost and dollar budget price it at the loser's own model. The estimate never appears in the row's token columns. An attempt refused before any 2xx, or ended by the provider's own error frame, is not charged. An attempt still running when the race ends is judged by what it had answered at that moment. A sequential streaming attempt whose TTFT probe fails behind a 2xx is charged the same way.

It applies to streaming chat completions with more than one candidate only, so in practice to hotel/ group traffic. A single-provider request, a non-streaming request and the multimodal endpoints all keep the ordinary sequential loop. The hedge_delay value is floored at 100ms whatever the settings API is given, so a 0s cannot fire every candidate at once. A candidate skipped without a request because its provider sat at its in-flight window still gets the bounded wait for a slot before the group answers all-busy: turning hedging on never makes a merely saturated group fail faster than leaving it off.

Every launched attempt appears in the per-attempt trail marked as hedged, so the fan-out is visible after the fact, and each one counts towards modelhotel_failover_attempts_total. An attempt abandoned in flight is recorded with no status and the exit as its cause: hedge_superseded when another candidate won, failover_timeout at the deadline, client_disconnect when the client left. A loser whose result had already arrived keeps its own verdict.

The cost is duplicate upstream load on every slow start: provider rate limits and capacity are consumed faster, and a backup that ignores cancellation can keep generating in the background. That is why it ships off. Turn it on only if slow first tokens hurt more than the extra load.

Once streaming begins, a stall watchdog monitors for silence in the SSE stream:

• Timeout: Configurable via stream_stall_timeout setting (default: 30s). If no data arrives within this window, the connection is terminated and the circuit breaker records a failure.
• Progressive extension: After 50 chunks, the effective stall timeout is multiplied by 3 (default: 30s × 3 = 90s) to tolerate tool-call pauses and long reasoning chains.
• Large frames: bytes arriving inside one long SSE line also count as activity: every 4 KiB of a line re-arms the watchdog. Once the line has been arriving for three stall windows it must also have averaged at least 64 KiB/s since it began, so a large frame on a slow but working link keeps going while an upstream trickling bytes to hold the stream open still stalls.
• Disable: Set stream_stall_timeout to 0s to disable the watchdog.

The resolved_model_id field in request logs records the actual upstream model ID used (e.g. z-ai/glm-4.6), which may differ from the requested hotel/ name (e.g. hotel/glm-4.6). For non-hotel requests where the requested and resolved model are the same, this field is NULL.

Each request produces one request_logs row. Its flat columns carry the final attempt's data, and every attempt before it is kept in that same row's attempts column:

• failover_attempt says how many attempts were made (0-indexed), so a request that succeeded on the first try has failover_attempt = 0
• resolved_model_id records the actual upstream model ID used
• attempts holds one element per attempt, in order, including the candidates the circuit breaker refused before any request was made. See The per-attempt trail for the element shape

So the table still has exactly one row per incoming request, and that row carries the whole failover chain rather than only its outcome.

Attempts are logged as they happen too (slog text format shown):

level=INFO msg="proxy: failover triggered" attempt=1 provider=NanoGPT provider_id=... status=429 rate_limit_class=saturated
level=INFO msg="proxy: failover backoff" backoff=147ms attempt=2

During proxy requests, token usage is tracked against the virtual key used for authentication. Token counts (prompt tokens, completion tokens, and total tokens) are extracted from the upstream provider's response and recorded in the request_logs table. This enables usage tracking and quota enforcement per virtual key.

The proxy includes a circuit breaker that prevents sending requests to models that are consistently failing, giving them time to recover.

What a circuit is keyed on: one circuit per (provider, resolved upstream model) pair - the model id actually sent upstream, never the hotel/ group alias. A model that fails for its own reasons (a plan that excludes it, a retired id, a per-model refusal) darkens only itself; its healthy siblings behind the same provider keep serving. The provider as a whole is a derived verdict, never a circuit that anything charges directly: see Provider-wide verdict.

What is charged, credited, and ignored: 5xx, 401/403, a 429 the classifier could not read, and timeouts are charged as failures. A 402, and a 429 read as exhausted, open the circuit on that one answer instead of waiting for the threshold. A 429 read as saturated is a no-op, neither charge nor credit: a provider at its concurrency ceiling is alive, and benching it wastes the slots that are busy serving. So are 404 and 499, the two failover-eligible statuses that say something about this request rather than about the provider; crediting them would erase real 5xx history.

A non-2xx status that does not trigger failover (a 400 parameter rejection, a 422) is recorded as alive: the provider processed the request and refused it, which is a health signal but not a serve, so it does not satisfy the recent-success fallback the 429 classifier uses. A 2xx records nothing at header time; its verdict waits until the body has been read, because a provider answering {"choices":[]} to everything must still be able to open a circuit.

A success resets that model's circuit only: a working model says nothing about a failing sibling, and crediting the sibling is how a broken model stays in rotation forever.

Each model circuit is a three-state machine:

// internal/failover/circuitbreaker.go (abridged)
type CircuitBreaker struct {
    mu sync.RWMutex
    // provider UUID → resolved upstream model id → circuit state
    circuits map[string]modelCircuits
    names    map[string]string    // provider UUID → operator's name
    settings SettingsReader       // runtime threshold, cooldown, span, backoff
    quota    QuotaAdvisor         // quota reset deadlines for pinning; nil disables
    onOpen   func(providerID uuid.UUID)

    Threshold         int           // Default: 5 consecutive failures
    SpanModels        int           // Default: 2 open model circuits to indict the provider
    Cooldown          time.Duration // Default: 60s
    HalfOpenMaxProbes int           // Default: 1 success to close
}

// internal/failover/model_circuits.go (abridged)
type modelCircuits map[string]*circuit

type circuit struct {
    state            State         // Closed, Open, or HalfOpen
    consecutiveFails int
    openedAt         time.Time     // When transitioned to Open
    halfOpenProbes   int           // Successful probes in HalfOpen
    cooldownOverride time.Duration // Quota pin, when one governs this circuit
    failedProbes     int           // Half-open probes that failed since the circuit last closed
    cooldownBackoff  time.Duration // The cooldown those failures earned: base doubled per probe, capped
    lastCharged      time.Time     // Orders eviction above the per-provider cap
    lastSuccess      time.Time     // Backs the recent-success 429 fallback
    pinSource        string        // "advisor", "reserve", "response" or "account"; empty when unpinned
    probeSeed        float64       // Spreads a response pin's probes
    lastCause        string        // Last verdict, its upstream status and when (observability only)
    lastStatus       int
    lastAt           time.Time
    opens429Streak     int         // Consecutive 429-caused opens (see Probe backoff)
    open429WindowStart time.Time
    opens            int           // Opens in the 24h window, for circuit_breaker.unstable
    openWindowStart  time.Time
    unstableReported bool
}

The map is capped at 256 circuits per provider. When a provider reaches it, the least recently charged closed circuit is evicted to make room; an open or half-open circuit is never dropped, because evicting one would silently restore a model the breaker has decided is broken. The cap is therefore a soft one: a provider holding 256 circuits that are all open has no eligible victim, so the map keeps every one of them and is allowed to grow past 256 rather than un-darken a broken model.

State Transitions (per model circuit):

A half-open circuit is not a single-request gate: once the cooldown has elapsed the circuit simply stops blocking, so every request that arrives before the first probe settles is admitted. The first outcome decides for all of them, closing the circuit on a success or reopening it with a fresh cooldown on a failure. In practice the probes are real traffic, not synthetic health checks, so at any sane request rate that window is short.

A provider is skipped for every model when either of these holds:

• a quota pin that speaks for the whole account is in force: one the quota advisor measured (pin_source advisor, or reserve when the quota reserve drew the line), or one taken from a 429 that refused the account itself (pin_source account, see "A refusal about the account" under Quota-pinned cooldowns). A plain response pin is one model's own refusal and darkens only that model, or
• at least circuit_breaker_span_models distinct model circuits are open and still inside their cooldown.

Nothing stores that verdict; it is recomputed from the circuits on every read, so it cannot go stale against a cooldown that elapsed a moment ago. A circuit owed a probe blocks nothing, so it counts towards neither the verdict nor the list of models beside it.

That also means the verdict holds for as long as enough circuits keep blocking. With probe backoff on, a circuit whose probes keep failing stays inside its cooldown for up to circuit_breaker_backoff_max at a stretch, so a provider indicted by two such circuits can stay skipped, healthy siblings included, until the first of them comes up for a probe, rather than clearing every circuit_breaker_cooldown. The 15-minute default ceiling is chosen with that in mind.

The span default of 2 is the smallest number that requires corroboration: one model refusing is evidence about that model, two models refusing is evidence about the provider. Setting it to 1 restores the older behavior, where the first open circuit sidelines the whole provider, and is the escape hatch if you want that back.

The dashboard reads it the same way. The red number in the Failover nav badge counts providers the breaker is skipping - the derived verdict, read off the provider_open field of the detail rows - and the amber number beside it counts providers that hold an open or probing circuit but are still routing. The status endpoint's own closed / half_open / open counts are per-circuit-state tallies of the most degraded circuit per provider, which is a different question: a provider serving three models with one dark appears under open there and in the amber number here.

Every circuit remembers its last verdict: the reason it was last charged, credited, pinned or released, the upstream HTTP status behind that (0 when no response was seen) and when it landed. The reason is always a fixed phrase chosen by the gateway, never text copied from a provider response, so it can carry neither a prompt echo nor a credential into a status page or an alert.

A request log that kept only its terminal attempt hides the interesting half of a failover: a provider's 429 on attempt 0 of a request another provider then served left no trace at all. So a request log row that reached a candidate carries attempts: one element per failover attempt, in order, with the provider, the resolved model, the upstream status it reached, its error kind, up to 160 characters of the sanitized and credential-masked upstream error, the rate-limit phrase it matched, its duration and TTFT, whether it was a hedged probe, and what it did to the circuit (breaker: charge, noop, success, alive, skipped, disabled).

Candidates the breaker refused at resolve time lead the trail as attempt: -1, breaker: skipped, so the operator also sees who was never asked and why. A hedged attempt still in flight when the race ended is recorded too, with no status and the exit as its cause (hedge_superseded when another candidate won, failover_timeout at the deadline, client_disconnect when the client left): it was launched and abandoned, and the fan-out it represents is what the per-provider failover counter reads off the trail. A loser whose result was already queued when the race ended keeps its own verdict. The column is stored sorted by attempt index, skips first.

Three surfaces read it. The request detail dialog renders it as a chain: each row carries its status or no response badge, and beneath it the breaker verdict (breaker: strike, reset, untouched, alive or disabled, with a shield glyph) and the error kind as a labelled classification (client disconnected, provider saturated, failover timeout, ...) with the raw kind in its tooltip; the provider's own error text follows in monospace only when it says something the badges and labels do not. On a Front Desk fleet the Members table's Circuits column shows each member's open circuits with the same causes (see High Availability). And GET /api/logs?attempt_provider_id=<id>&attempt_status=429 lists every request in which that provider answered 429, whoever served it in the end (see the API reference). The trail is also what keeps the 429 phrase table honest: a phrase that has matched no attempt in 90 days is named in a daily app-log report.

On the Failover page each entry now carries a chip read from its own circuit: live, busy (the circuit is closed but its last verdict was a saturated 429 inside the last minute, so requests still route here and spill to the next entry when it is full), open, probe due, or pinned (held by a quota pin). The chip's tooltip names the last cause, the upstream status behind it and when it landed, and the card's footer says 2 of 3 entries live, or all entries dark in red with the earliest retry. A member from before circuits[] falls back to the row and open_models, so a mixed-version fleet renders every entry either way.

GET /api/failover-groups/circuit-breaker-status?detail=1 lists every circuit behind a provider row under circuits[], sorted by model, each with its own state, consecutive_fails, opened_at / cooldown_ms / next_retry_at (while the wait is enforced), quota_pinned / pin_source / backed_off / failed_probes (the overrides governing that circuit), and last_cause / last_status / last_at. The open entries in circuits[] are exactly open_models, which stays what the dashboard keys on; a member from before this field simply omits it. The circuit_breaker.open event and the model state=...→open log line carry the same cause and status, and the event's message names the cause, so an outbound alert reads Provider NeuralWatt circuit breaker: open for model glm-5.3: upstream status 429 (saturated) rather than only open.

Blocked candidates are filtered out during provider resolution:

// internal/proxy/resolve.go:buildFailoverCandidates
if cbEnabled && h.circuitBreaker.IsOpen(prov.ID, prov.Name, m.ModelID) {
    retryAt, pinned, dated := h.circuitBreaker.BlockedUntil(prov.ID, m.ModelID)
    skips.note(retryAt, pinned, dated) // dates an all-skipped group's 429
    // ... the skip is also kept for the per-attempt trail
    debuglog.Info("resolve: skipping candidate: circuit breaker open", "provider", prov.Name, "model", m.ModelID)
    continue // Skip this candidate
}

IsOpen skips a candidate when its own model circuit is open, or when the provider-wide verdict is open. So a group member whose model is dark is never tried at all, while its sibling behind the same provider stays in rotation until the span is reached.

Client Disconnections:

An abandoned attempt does NOT count as a provider failure: the client hung up, or a hedge sibling won the race. Everything else that stops a request before it completes is charged once per candidate attempt (after any transient transport retries are spent): a real transport error, the transport's own header timeout (the provider accepted the connection and never answered), and the gateway's own failover or retry deadline expiring while the caller is still waiting, which is the same stall the TTFT probe and the body readers already charge:

// internal/proxy/proxy_upstream.go
if !abandoned {
    // No status: the request never completed, so there is none.
    h.chargeBreaker(st, candidate, 0, "upstream request failed")
}

// internal/proxy/breaker_outcome.go:chargeBreaker
h.circuitBreaker.RecordFailure(candidate.provider.ID, candidate.provider.Name, candidateModelID(candidate), failover.Cause{Status: status, Reason: reason})

An abandoned attempt is a user-side cancellation, not a provider health issue.

All of these settings are runtime-configurable and take effect immediately. Most have UI controls in the Settings page, under Circuit Breaker & Failover, which is laid out in four groups:

• Failover: the kill switch, failover-on-429, the failure threshold, the model span, the cooldown, the quota-pin ceiling and the probe-backoff ceiling
• Hedging: hedging_enabled and hedge_delay (see Request hedging), with the load trade-off spelled out beside the toggle
• Rate Limit (429) Handling: classification, the saturation wait, the recent-success window, open-on-exhaustion, the 429 exhaustion status and the last-candidate 5xx retry
• Adaptive Concurrency: the in-flight limiter and its grow / forget windows

ttft_timeout and stream_stall_timeout live in the Proxy section instead, and circuit_breaker_pin_probe_interval has no control of its own. Every key here can also be set via PUT /api/settings.

Without pinning, a provider that has exhausted its quota is re-probed every circuit_breaker_cooldown until the window resets. On a weekly plan that is thousands of doomed requests, each one a real user request that fails first. Quota pinning stretches an already-open circuit's cooldown out to the provider's actual reset deadline, clamped to circuit_breaker_quota_pin_max and jittered so a fleet does not stampede the provider at the same instant.

Pinning normally only changes the cooldown of a circuit that has already opened, and a wrong or stale quota reading can then delay a retry but never sideline a healthy provider. There is one deliberate exception, because circuit state is in-memory and quota readings are not: a measured exhaustion reading also opens a circuit of its own (quota pin seeded (advisor)). After a restart, on a member that has never served the provider, and for a provider nothing has requested since its window went spent, every circuit starts closed while the stored snapshot already says the account is out, so the quota badge reads spent and the gateway keeps routing there until two real refusals reopen a circuit. The advice pass closes that gap within one poll instead, and spends no request doing it, so a badge that reads spent and a pinned provider now agree.

The seeded circuit is one per provider, listed under the model id (account quota), and it carries the advisor pin, which speaks for the whole account: the derived provider verdict darkens every model of that provider off it, including models nothing has ever routed to. It is bounded exactly as every other pin is, only a reading with a datable future reset seeds anything, circuit_breaker_quota_pin_max caps it, circuit_breaker_cooldown floors it, and setting the ceiling to 0s stops seeding along with the rest of pinning. It is skipped entirely when the provider is already held dark account-wide. Releasing the pin retires the circuit with it, since nothing ever routed to it and no probe would ever close it. A reading that stops being usable (the snapshot goes stale, the payload can no longer be assessed, the provider disappears) is not a recovery and does not lift the pin while it is in force, but it is also not evidence the window is still spent: once the pin elapses, the next advice pass retires the circuit and the provider is routed to again.

These behaviors are worth knowing before you reach for these controls:

• A pin ends by itself when the provider recovers. A pin is stamped on at the moment the circuit opens, but it is not a commitment to wait that long. The quota poller refreshes its picture every quota_refresh_interval_min minutes (5 by default), and any provider that is no longer out of quota has its account-level pins (advisor, reserve and account) lifted on the spot. A plain response pin is one model's own refusal (a model outside the plan, say), which a healthy account reading does not contradict, so it keeps its own timer and probe interval. The circuit falls back to circuit_breaker_cooldown (or to its probe backoff, if its probes had been failing before the pin) and, if that has already elapsed, goes half-open and probes on the next request. So an operator who tops up a spent plan gets the provider back within one poll interval instead of waiting out a pin that could run to the 24h ceiling. Lifting a pin only ever shortens a wait: it does not close the circuit, and the probe still has to succeed before traffic resumes. Look for circuit-breaker: quota pin released (provider no longer exhausted) in the logs.

  A pin is released only on positive evidence of recovery: a refresh that actually succeeded, which read a snapshot fresh enough to trust and understood it well enough to say the window is no longer spent. Everything short of that leaves the pin exactly where it is. A refresh that could not read the database, a provider whose quota endpoint has been failing long enough for its last snapshot to go stale, a snapshot whose own most recent refresh attempt failed (the last good payload is kept on file for the dashboard, so it can still look fresh even though nothing since has confirmed it), and a payload the gateway cannot interpret all say nothing about provider health, and treating any of them as recovery would put a still-exhausted provider back into rotation precisely when quota reporting is broken.

  In a fleet this judgement is made on the same evidence everywhere: members mostly act on snapshots Front Desk distributes from the primary rather than on their own polls, and a snapshot travels with the record of whether the refresh behind it succeeded, so a member reaches the same verdict on it as the primary does.

• Turning quota polling off releases pins already in force, too. Setting quota_refresh_interval_min to 0 ("Disabled") stops the feature acting on that node completely, not just from that moment on. It stops new pins, because there is no fresh snapshot to advise from, and it releases every pin currently held, because nothing is left running that could ever report a recovery, so a pin left standing would be served out to the 24h ceiling on evidence you deliberately stopped collecting. The rule above still holds while the poller is running: absence of evidence keeps a pin only for as long as the gateway is still looking. Look for circuit-breaker: quota pin released (quota polling disabled) in the logs, one line per released pin.

• Turning pinning off releases a pin that is already in force. The ceiling is the switch: circuit_breaker_quota_pin_max set to 0s turns pinning off, and it is re-read on every check rather than only when a circuit opens, so an operator whose provider has been sidelined for hours can recover it by zeroing the limit. Allow up to about 30 seconds for the change to reach the proxy: settings are cached with a 30 second TTL, so the release is not instantaneous. An absent key, as opposed to a stored zero, means the 24h default.

• A refusal about the account darkens the whole provider at once. When a 429 body says the account itself has nothing left (OpenAI's insufficient_quota, MiniMax's balance code), the pin is marked account-wide and the provider verdict opens on that single answer, so no sibling model is sent to draw the same refusal. While that pin's hourly probe is out the siblings are served again, and one the account still refuses takes its own account pin, which darkens the provider afresh; each such pin then probes on its own interval, so a provider is back once the last of them has probed. Turning pinning off releases an account pin like any other. A plan that excludes one model (Z.ai's no resource package) stays a per-model pin, and the ordinary span rule still applies to those. Only a 429 is read this way: Anthropic states its low balance on a 400, which the classifier never sees.

• A pin the provider's own response produced probes itself. When no quota endpoint exists for a provider (OpenAI, Anthropic, Google, xAI), the pin can only come from the 429 body, and nothing measures when that window ends; a topped-up account would otherwise stay dark until the ceiling. Such a circuit goes half-open every circuit_breaker_pin_probe_interval (default 1h, jittered by up to 5%) and admits requests until one reports: a refusal re-pins for another interval, a success closes the circuit. 0s turns this off. Pins the quota advisor measured are unaffected, and so is a circuit whose probe backoff already reaches the pin: the floor that keeps a pin from making the breaker more aggressive applies here too, so such a circuit keeps its backoff until an ordinary probe succeeds or an operator resets it.

Between them, these give you three ways out of a long pin, from least to most blunt: wait for the poller to notice (automatic, up to one poll interval), reset the one provider whose circuit you want back (immediate, see Resetting a circuit), or turn pinning off fleet-wide (releases every pin at once). Only the reset also clears a probe backoff; the other two release the pin into whatever backoff the circuit had earned. Disabling quota polling altogether releases every pin as well, but it also blinds the quota dashboard, so zero circuit_breaker_quota_pin_max when pinning is the only thing you want stopped.

By default a provider is routed to until its window is spent and the provider says so. A quota reserve on the provider (edit modal, a slider in 10% steps from 0 to 90, where 0 drains fully; the slider appears only for the provider types listed below, since nothing else reads it, and stays while a reserve saved earlier on another type waits to be cleared; the providers list shows a yellow piggy-bank icon while one is set on a type that reads it) keeps that share of the provider's longest window back for use outside the gateway: an IDE plugin or a chat client on the same subscription. The line is drawn on the weekly window for Z.ai Coding Plan, Kimi Code, MiniMax (each model's weekly window) and NanoGPT, and on the monthly window for OpenCode Go; the edit modal and the piggy-bank tooltip name the window. Shorter windows (a 5-hour window, NanoGPT's daily counter, OpenCode Go's week) refill on their own and stop routing only when spent. Once the longest window has consumed 100 - reserve percent of itself, the quota refresh advises the provider exactly as a spent window would, so the same pin holds it dark until that window resets and the same refresh releases it once the reading drops back under the line. The pin reports pin_source reserve rather than advisor, the failover entry chip reads reserved rather than pinned, and the seeded-circuit log line says reserve reached rather than exhausted: the quota is still there, just kept back, which is a different state from an account that has nothing left. Only windows the breaker judges on their own count: Z.ai Coding Plan's token windows (not its MCP call window), Kimi Code, OpenCode Go, MiniMax and NanoGPT's input-token counters (a NanoGPT plan with overage enabled keeps serving past its cap, so the reserve is the only line that stops routing to it). NeuralWatt is a balance provider that serves into overage, so a reserve leaves it alone, and an undated window (a prepaid balance) never places a pin. 0 is the default and drains fully. The value rides config sync like every other provider field, and modelhotel_provider_quota_reserve_ratio carries it to Prometheus.

The half-open probe is not a synthetic health check: IsOpen hands it to the next real request for that model, and if the model is still broken that request fails before failover moves on (or, when the model has no healthy sibling in its group, fails outright). At the default 60s cooldown a model that stays broken all day therefore wastes about 1,440 real requests, one per cooldown, and circuit_breaker.unstable only reports that waste. Probe backoff removes most of it: every probe that fails doubles the cooldown before the next one, and a probe that succeeds closes the circuit and resets the count. At the defaults the waits run 1, 2, 4, 8 minutes and then hold at the circuit_breaker_backoff_max ceiling of 15 minutes, so the same broken model wastes a hundred or so requests a day, and a model fixed upstream is back in rotation within a quarter of an hour with nobody touching anything.

The ceiling is deliberately not longer. A probe is the only way a model with no healthy sibling ever gets tried again, so the ceiling is also the longest such a model can stay unreachable after it has actually recovered; and the provider-wide verdict holds for as long as its blocking circuits do, so it bounds how long a partially broken provider's healthy models stay skipped as well. Raise it if you would rather retry less often and can wait longer for a recovery to be noticed.

What counts is a failed probe, not an open: a circuit that closes on a successful probe and is later charged back to threshold by fresh failures is a new incident and starts again from the base cooldown. That is deliberate. Doubling on every open would slow the recovery of a model that had two real, separate outages in one day, which is exactly the model an operator wants back promptly.

Three things are worth knowing:

• The longest cooldown in force governs, and a pin is floored at the backoff. (One exception: a pin inferred from a response rather than measured by the quota advisor is served as circuit_breaker_pin_probe_interval instead of its full length, and its refused probes do not grow the backoff; see the quota pinning section.) A circuit can carry a quota pin and a backoff at once; it serves whichever reaches further, and quota_pinned and backed_off each say whether their override is in force. A pin is floored at the cooldown in force when it is stamped, after the circuit_breaker_quota_pin_max ceiling has been applied, so a quota reading that says the window resets in three minutes does not pull a four-minute backoff in, and neither does a low pin ceiling; only a reading further out than the backoff pins, and then the pin is the longest. When a pin is lifted early, the circuit falls back to its backoff, not to the base cooldown: a provider that recovered its quota is still a provider whose probes failed. Raising circuit_breaker_cooldown past a backoff stamped earlier makes the new cooldown govern, and the row stops claiming a backoff.
• A circuit that only ever reopens on 429s may back off further. From the third open inside 24 hours in which every open was caused by a 429 (unclassified or exhausted; a 402 or a 5xx open resets the run), the backoff ceiling is lifted from circuit_breaker_backoff_max to circuit_breaker_quota_pin_max, since such a circuit is spending a request per cooldown against a window that resets in hours. A successful probe, a manual reset or an affirmative quota recovery clears the run. With backoff switched off (circuit_breaker_backoff_max at 0s) nothing escalates.
• Turning backoff off releases a backoff already in force. The ceiling is the switch: circuit_breaker_backoff_max set to 0s turns backoff off, and like the quota-pin ceiling it is re-read on every check, so zeroing it puts every backed-off circuit back on circuit_breaker_cooldown within the settings cache TTL (about 30 seconds). The failed-probe count keeps accruing while backoff is off, and once it is back on the next failed probe applies the backoff every counted failure has earned; the status API reports the count whether or not it governs anything. A ceiling at or below circuit_breaker_cooldown leaves backoff with nothing to add and is the same as off. An absent key, as opposed to a stored zero, means the 15-minute default.

Every surface derives from the same predicate. The status API's cooldown_ms and next_retry_at are the backed-off values, backed_off says that is why, and failed_probes says how many times. The circuit_breaker.open event carries all four and, when the backoff is the value in force, says so in its message (... (backing off after 3 failed retries, next retry in 8m)), which is what an outbound alert renders; a circuit held longer by a quota pin gets no such suffix, since then it is quota and not failed retries that holds it; the log line carries the flag, the count and the cooldown; and a quota pin released log line reports the cooldown the circuit actually falls back to. Resetting a circuit (below) clears the count with everything else. The Prometheus gauge is unchanged: it buckets by state and cannot tell a 60-second circuit from a fifteen-minute one.

The circuit breaker publishes real-time events via the SSE event bus:

circuit_breaker.unstable reports and never retires, which is the whole of its design. Retiring a model means asserting the provider no longer serves it, and only a request refused by name establishes that; a model failing that way is already handled by the model-gone path on its first refusal. What is left for this event is the model that fails some other way over and over (timeouts, 5xx, a provider having a bad week), and disabling that model would take a working one out of routing on evidence that says only "upstream is unwell". It reports at most once per model per 24 hours: the window keeps running after it fires, so a permanently broken model does not report every third open.

It is off by default in the alert catalogue, so it sends no outbound (Apprise) notification until an operator turns it on under Settings, Alerts. That default covers outbound alerting only: the dashboard's live event feed is not catalogue-gated, so the event still appears there as it happens. It is off by default because an instance upgrading with a long-flaky provider would otherwise notify on the third open about a condition its operator already lives with.

The transition into Half-Open itself (an open circuit whose cooldown has elapsed, now allowing a probe through) is not published as an event: it is a transient internal state, surfaced only via the circuit-breaker status API's half_open count, not the alert/SSE event stream.

These events appear in the real-time sidebar and dashboard.

circuit_breaker.open and circuit_breaker.closed carry the same metadata block:

next_retry_at is the retry deadline, not the provider's quota reset time: under a pin it is the clamped and jittered pin, so a weekly quota that resets days out still yields a next_retry_at at the circuit_breaker_quota_pin_max ceiling, and under a backoff it is the doubled cooldown. It is the same value the circuit-breaker status API publishes under that name.

// internal/failover/circuit_events.go:publishEvent
meta := map[string]any{
    "provider_id":       providerID.String(),
    "provider":          providerName,
    "model":             model,
    "state":             state,
    "cause":             c.lastCause,
    "status":            c.lastStatus,
    "pin_source":        cb.pinSourceForWith(c, r),
    "provider_open":     providerOpen,
    "consecutive_fails": c.consecutiveFails,
    "quota_pinned":      pinned,
    "backed_off":        backedOff,
    "failed_probes":     c.failedProbes,
}
if state != "open" || !providerOpen {
    meta["model_id"] = model
}
if state == "open" {
    meta["cooldown_ms"] = cooldown.Milliseconds()
    if pinned || backedOff {
        meta["next_retry_at"] = c.openedAt.Add(cooldown).Format(time.RFC3339)
    }
}
msg := breakerEventMessage(providerName, state, model, providerOpen, c.lastCause)
if state == "open" && backedOff && cooldown == c.cooldownBackoff {
    msg += backoffSuffix(cooldown, c.failedProbes)
}
events.Publish(events.Event{
    Type:     "circuit_breaker." + state,
    Severity: cb.severityForState(state),
    Source:   "failover",
    Message:  msg,
    Metadata: meta,
})

The message names the model, because the event is about one model circuit, and the cause after it: Provider zai circuit breaker: open for model glm-4.6: upstream status 503. When that transition also flips the provider-wide verdict, the message says so as well - ... for model glm-4.6 (provider skipped): upstream status 503 - since that is the moment the remaining models leave rotation. Without the model in the sentence, a toast or an Apprise alert about one sidelined model reads as a whole provider going down.

A separate quota.schema_drift event fires when a provider changes the shape of its quota response (the set of key paths, not the values). It carries the added and removed paths and is alert-only: it never opens a circuit, never pins a cooldown, and never affects routing. See Alerting and the API Reference for its metadata.

A pin being lifted early is likewise not an event. Nothing transitions at that moment: the circuit is still open, it has simply gone back to the cooldown it would otherwise serve (its probe backoff if one is in force, else the ordinary one). The change shows up in the next circuit-breaker status poll (quota_pinned flips to false and next_retry_at moves in) and in the log line quoted above.

Waiting out a cooldown is not always what you want. If you have just fixed the provider yourself (rotated a dead key, restarted a local runtime, paid an overdue invoice) the breaker has no way to know, and you can force the circuit back into rotation:

One provider:

curl -X POST http://localhost:8080/api/failover-groups/circuit-breaker/$PROVIDER_ID/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

This is also the circular-arrow button ("Reset circuit breaker") beside each open or half-open member on the Failover page, so the common case needs no curl at all.

One circuit only (the provider's other models keep their charges):

curl -X POST "http://localhost:8080/api/failover-groups/circuit-breaker/$PROVIDER_ID/reset?model=glm-5.3" \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Every circuit behind one failover group, on this member:

curl -X POST http://localhost:8080/api/failover-groups/$GROUP_ID/circuit-breaker/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Across a fleet each member keeps its own breaker, so Front Desk offers the same group reset fanned out to every member: the Reset circuit breakers button on its Members page picks a group and asks each member with a stored token, reporting who cleared what and who could not be reached (POST /api/fleet/circuit-breaker/reset with {"group_id": ...}). Every manual reset, however scoped, logs one circuit-breaker: manual reset line per circuit with cause=manual reset.

Every provider at once:

curl -X POST http://localhost:8080/api/failover-groups/circuit-breaker/reset \
  -H "Authorization: Bearer $ADMIN_TOKEN"

Reset-all is deliberately API only, with no UI control. That is a decision, not an omission. Per-provider reset is the operation you want almost every time, and it names the provider you are recovering; a fleet-wide button sitting next to it is far too easy to reach for when a single circuit is the problem, and it discards evidence about every other provider at the same time. The bulk endpoint exists for the case it was written for: a fleet-wide upstream incident that tripped many circuits at once, cleared from a script or a runbook where the blast radius is spelled out.

Resetting clears the circuit outright: state, failure count, and any quota pin. The provider is tried again on the very next request, so if it is still broken the circuit simply reopens after circuit_breaker_threshold failures. Both endpoints are safe to call when nothing is wrong: an untracked or already-closed provider reports success and no change rather than an error. See the API Reference for the response shapes.

GET /api/failover-groups

Returns all failover groups with their entries and configuration.

Response (200):

{
  "groups": [
    {
      "id": "uuid-of-group",
      "display_model": "glm-4.6",
      "display_name": "GLM 4.6",
      "description": "Multi-provider failover",
      "group_enabled": true,
      "auto_created": true,
      "auto_disabled": false,
      "entries": [
        {
          "model_uuid": "uuid-1",
          "model_id": "z-ai/glm-4.6",
          "provider_id": "uuid-prov-1",
          "provider_name": "OpenRouter",
          "display_name": "GLM 4.6",
          "enabled": true,
          "model_enabled": true,
          "provider_enabled": true,
          "disabled_manually": false,
          "context_length": 200000,
          "owned_by": "z-ai"
        },
        {
          "model_uuid": "uuid-2",
          "model_id": "zai-org/glm-4.6",
          "provider_id": "uuid-prov-2",
          "provider_name": "NanoGPT",
          "display_name": "GLM 4.6",
          "enabled": false,
          "model_enabled": true,
          "provider_enabled": true,
          "disabled_manually": false,
          "context_length": 200000,
          "owned_by": "zai-org"
        }
      ],
      "total_tokens": 1234567,
      "created_at": "2024-01-15T10:30:00Z",
      "updated_at": "2024-01-20T14:45:00Z"
    }
  ],
  "last_synced_at": "2024-01-20T14:45:00Z"
}

enabled on each entry is the per-entry intent toggle; model_enabled / provider_enabled expose the underlying model and provider state the router also checks. An entry is effectively routable only when all three are true - the UI greys out and badges entries where they disagree. disabled_manually tells a model an operator switched off from one discovery disabled (the provider stopped listing it). On the group, auto_disabled is true when the group is off and no operator chose that (discovery's auto-disable or the dashboard's member-toggle cascade stamped it; see auto_disabled_at / floor_disabled_at above).

GET /api/failover-groups/{id}

Response (200): Same structure as single group in list response.

Response (404): failover group not found (plain text, as is every error body on these admin endpoints)

POST /api/failover-groups

Request:

{
  "display_model": "llama-3.1-8b",
  "display_name": "Llama 3.1 8B",
  "description": "Custom failover group for Llama 3.1 8B",
  "entry_ids": [
    "uuid-model-1",
    "uuid-model-2",
    "uuid-model-3"
  ]
}

Validation:

• display_model is trimmed and lowercased (1 to 128 characters) and must be unique
• display_name up to 128 characters, description up to 500
• Minimum 2 entries required
• Every entry_ids element must be a valid UUID of a model this instance has, with no duplicates

Response (201): Full group object (same as GET response).

Response (400): Validation error, plain text: at least 2 entries required for failover group, invalid entry_id: <id>, duplicate entry_id: <id>, unknown entry_id: <id>.

Response (409): A failover group for '<display_model>' already exists (auto-created from shared model). Edit the existing group instead.

PUT /api/failover-groups/{id}

Request:

{
  "display_name": "Updated Name",
  "description": "Updated description",
  "display_model": "glm-4.6-renamed",
  "group_enabled": true,
  "priority_order": ["uuid-3", "uuid-1", "uuid-2"],
  "entry_enabled": {
    "uuid-1": true,
    "uuid-2": false,
    "uuid-3": true
  }
}

All fields are optional. Only provided fields are updated. display_model renames the group (and so its hotel/ name): it is trimmed, lowercased and must not collide with another group. A priority_order replaces the member list wholesale.

Validation:

• priority_order entries must be valid UUIDs, the list must not be empty and must not repeat an id
• entry_enabled holds at most 100 entries
• A group that stays or becomes enabled needs at least one enabled entry
• Turning a disabled group on needs at least two routable members (entry, model and provider all enabled)

Response (200): Updated group object.

Response (400): Validation error, plain text, e.g. priority_order must not be empty, duplicate priority_order entry: <id>, at least one entry must be enabled for an active failover group, a failover group needs at least 2 enabled members (model and provider enabled) to be active.

Response (404): failover group not found

Response (409): A failover group for '<display_model>' already exists. Choose a different name.

Editing or deleting an auto-created group on a Front Desk fleet member drops the member's copy of the auto groups the primary last sent, so its config hash stops matching and Front Desk re-applies the primary's order on top of the edit (see High Availability). A no-op on the primary and on a standalone instance.

DELETE /api/failover-groups/{id}

Response (204): No content. A group that does not exist also answers 204 (the delete is a no-op); only a malformed id is refused (400).

POST /api/failover-groups/sync

Manually trigger failover group synchronization with current model discovery state.

Response (200):

{
  "deleted_groups": [
    {
      "display_model": "glm-4.5-air",
      "reason": "only 1 enabled provider (need 2+ for failover)",
      "provider_count": 1,
      "provider_names": ["OpenAI"]
    }
  ],
  "disabled_groups": [
    {
      "display_model": "my-custom-group",
      "effective_count": 1,
      "reason": "fewer than 2 routable members (need 2+ for failover)"
    }
  ]
}

deleted_groups is always present (null when nothing was deleted). purged_entries (group_display_model plus pruned_model_ids), disabled_groups (custom groups switched off below two routable members) and sync_errors are omitted when empty.

Per-model syncs (run automatically during discovery) additionally report membership changes as updated_groups entries (display_model plus removed_model_ids / added_model_ids); a newly created auto-group reports every member as added. These appear in the discovery diff's failover_updated_groups, not in this endpoint's response.

GET /api/failover-groups/by-model/{model_uuid}

Returns brief info about the failover group containing a specific model UUID.

Response (200):

{
  "id": "uuid-of-group",
  "display_model": "glm-4.6",
  "position": 2,
  "total_entries": 3
}

Response (404): model not in any failover group

GET /api/failover-groups/candidates

Returns all enabled models that can be added to failover groups.

Response (200):

[
  {
    "model_uuid": "uuid-1",
    "model_id": "z-ai/glm-4.6",
    "provider_id": "uuid-prov-1",
    "provider_name": "OpenRouter",
    "display_name": "GLM 4.6",
    "context_length": 128000,
    "owned_by": "openai"
  }
]

Failover group lookups are cached with a 5-minute TTL:

// internal/failover/cache.go
const failoverCacheTTL = 5 * time.Minute

type failoverCacheEntry struct {
    group     FailoverGroup
    expiresAt time.Time
}

var failoverByModelCache = make(map[string]failoverCacheEntry)

Cache Operations:

Cache Invalidation:

The cache is invalidated on any mutation. A write never installs its RETURNING row: two concurrent writes can finish in reverse order and an install would hold the older row for the TTL, so the write drops the key and the next reader refills. Every invalidation advances the cache generation; a read-through that captured the generation before its SELECT installs nothing if it has moved.

// internal/failover/failover.go:UpsertWithConfig
func (r *Repository) UpsertWithConfig(...) (*FailoverGroup, error) {
    // ... INSERT/UPDATE query ...
    InvalidateFailoverCacheKey(fg.DisplayModel)  // the next reader refills
    return fg, nil
}

// internal/failover/failover.go:Delete
func (r *Repository) Delete(ctx context.Context, displayModel string) error {
    _, err := r.pool.Exec(ctx, `DELETE FROM model_failover_groups WHERE display_model = $1`, displayModel)
    InvalidateFailoverCache()  // Clear entire cache
    return err
}

Cache Warming:

On startup the cache is pre-populated with the rows listed at a captured generation, so a write that lands during the List is not undone by the warm:

// cmd/server/startup.go:warmCaches
gen := failover.CacheGen()
groups, err := deps.failoverRepo.List(ctx)
// ...
failover.WarmFailoverCacheAt(groups, gen)

Transparent failover and hotel routing work together seamlessly:

1. Client requests hotel/glm-4.6
2. Hotel routing resolves the failover group and selects the first provider (highest priority)
3. That provider returns a retriable error (5xx, 429, 401/403, 404, or timeout) → transparent failover kicks in
4. Proxy tries the next provider in the group's priority order
5. Retries continue sequentially through provider-b, provider-c, etc.
6. If all providers in the group fail, the client gets the last failure's status: the last upstream HTTP status with a gateway-written message, 502 for a transport or TTFT-probe failure, 504 when the deadline passed, or 429 with Retry-After when every provider was busy (see How It Works)

This means hotel routing provides ordered preference (which provider to try first), while transparent failover provides exhaustive sequential retry - together they give you the best chance of a successful request.

"Failover group disabled" Error:

• Check group_enabled flag in group configuration
• A custom group is switched off automatically once it has fewer than two routable members
• Manual intervention required: re-enable after adding providers

"No available provider" Error:

• All entries in priority order are disabled (entry_enabled: false)
• All providers have open circuit breakers
• API key decryption fails (wrong MASTER_KEY)

Circuit Breaker Opens Frequently:

• Check provider health (API status, rate limits)
• Increase circuit_breaker_threshold (default: 5)
• Reduce circuit_breaker_cooldown for faster recovery (default: 60s)

Auto-Created Group Disappeared:

• Expected behavior: auto-created groups need 2+ enabled model rows sharing the base name, and one that drops below two is deleted by the next sync
• Add another provider with the same base model name and it is recreated
• Or use direct provider routing (e.g. OpenRouter/z-ai/glm-4.6) instead

The /metrics endpoint carries the failover story as counters, so a rerun of a bad afternoon shows up as rates rather than as a reset loop reconstructed from log lines:

model on the 429 and opens counters is the provider-side model id; modelhotel_requests_total carries the name the client asked for (hotel/<group> for group traffic), the same divergence the retirement-probe counter documents. No series carries a virtual key, a request id or any content.

Enable debug logging via DEBUG_LOG=true environment variable:

Output is slog's text format by default (set LOG_FORMAT=json for one JSON object per line). DEBUG_LOG_SCOPES takes a comma-separated list of scope prefixes (the text before the first : in a message, e.g. resolve, proxy, circuit-breaker) to raise just those to Debug:

level=DEBUG msg="resolve: failover group found" model=glm-4.6 entries=3 enabled=true
level=DEBUG msg="resolve: building candidates from failover group" model=glm-4.6 priority_order_count=3
level=INFO  msg="resolve: skipping candidate: circuit breaker open" provider=NanoGPT model=z-ai/glm-4.6
level=DEBUG msg="resolve: hotel model resolved" model=glm-4.6 candidates=2 decrypt_failures=0 breaker_skips=1
level=INFO  msg="proxy: failover triggered" attempt=1 provider=OpenRouter provider_id=... status=502 rate_limit_class=unknown
level=INFO  msg="proxy: failover backoff" backoff=147ms attempt=2
level=WARN  msg="circuit-breaker: model state=closed→open" provider=OpenRouter provider_id=... cause="upstream status 502" status=502 consecutive_failures=5 cooldown_ms=60000 quota_pinned=false backed_off=false failed_probes=0 model=z-ai/glm-4.6

The circuit-breaker lines name the model whose circuit moved, and providers appear by name; the provider UUID rides alongside as provider_id. The other transitions carry the same attributes under circuit-breaker: model state=open→half-open (cooldown elapsed), ... state=half-open→open (probe failed) and ... state=half-open→closed (probe succeeded).

Batch Queries:

Provider selection uses batch queries to minimize DB round-trips:

// Single query for all models
models, err := h.modelRepo.GetByIDs(ctx, enabledModelIDs)

// Single query for all providers
providers, err := h.providerRepo.GetByIDs(ctx, providerIDs)

Cache Hit Rate:

Failover group cache (5-minute TTL) should achieve >95% hit rate under normal load. Monitor cache misses in debug logs.

Circuit Breaker Overhead:

Circuit breaker state checks are O(1) with RWMutex-protected map lookups. The fast path (Closed state) uses only read locks.

• API Key Encryption: Provider keys are AES-256-GCM encrypted with Argon2id key derivation. Decryption happens in-memory only during request processing.
• Never Log Keys: Decrypted API keys are never logged or stored in request_logs.
• Virtual Key Hashing: Client virtual keys are SHA-256 hashed; only the hash is stored and compared.

• Configuration - Environment variables, database settings, runtime configuration
• Model Discovery - How models are discovered and failover groups are auto-generated
• Security - Provider key encryption and virtual key hashing
• API Reference - Proxy and admin API endpoints with examples
• Home - Architecture overview and feature summary