🇬🇧 English · 🇫🇷 Français

Applies to Hydranos 4.4.

This page is for anyone who wants to know whether Hydranos is healthy and what it is saying. It covers the health endpoints and the startup screen, the Prometheus page, the log outputs, the Logs, Overview and Benchmark tabs, the line test, tracker health, the anomaly report and the Linux memory diagnostics. For each one it also says what is not measured in 4.4, so you know where your monitoring has blind spots.

Starting, stopping, backups and updates are on Operating Hydranos.

flowchart LR
    E["engines race, hoard, extras"] --> T["tracing events"]
    W["workers, API, announcer"] --> T
    T --> C["console: docker logs or journald"]
    T --> F["hydranos.log beside default.toml<br/>rotated at 128 MiB, 4 old files kept"]
    T --> R["in-memory ring, 2000 lines"]
    R --> LT["Logs tab, GET /api/logs<br/>and /api/logs/stream"]
    S["bench sampler every 5 s<br/>every engine"] --> B["bench.db in data_dir"]
    I["iperf3 line test<br/>Run now or interval_secs"] --> B
    B --> BT["Benchmark tab and Records card"]
    E -->|"counters already kept"| M["GET /metrics<br/>rates, peers, announces, limits, memory"]
    M --> P["Prometheus"]
    E --> H["GET /health<br/>store and listeners"]
    E --> HS["health scan every 5 min"]
    HS --> A["GET /api/health/anomalies<br/>and the MCP health tool"]

{"checks":{"engines":[{"blocked":null,"held":false,"id":"race","listening":true,"online":true,"torrents":42}],"store":"ok"},"problems":[],"status":"healthy","uptime":86400.0,"version":"4.x.y"}

checks.store can also be busy (another reader holds the store at that moment): that is not a problem. An engine kept off the network on purpose (HYDRANOS_ENGINE_NET=0) is not expected to listen. An engine blocked by the kill switch makes the instance degraded, never unhealthy, even when every engine is blocked: a restart changes nothing until the config does (see Managed WireGuard).

In rescue mode /health answers the plain text ok, not JSON (HTTP 200). Rescue mode is described on Operating Hydranos.

Port 8199 (api_port) opens in the first second, before the catalogue is loaded; the log line is hydra API listening (starting) (with addr=). A port already taken therefore fails at once. Until the load ends, only these routes answer:

• / and /static/*: the web UI shows a startup screen (HYDRANOS, a phase line such as Restoring state…, Connecting to the network…, Opening the database… or Starting…, a bar and <restored> / <total> torrents), polls /api/startup every 500 ms and fades into the normal UI when ready turns true;
• /health (starting), /metrics (reduced set), /api/startup, and GET /api/setup.

Everything else — the rest of the API, login, the qBittorrent shim /api/v2/*, /mcp — answers 503 with Retry-After: 5 and {"error":"Hydranos is starting (<phase>): <restored> of <total> torrents restored","startup":{…}}. First-account creation (POST /api/setup) is refused too: the startup router only routes GET /api/setup until the load ends. When the load ends, the log says hydra API ready.

phase goes loading → connecting → opening_store → starting_workers → ready (or rescue). total only counts the engines that have started reading their records, so the bar can step back once per engine. Once ready, /api/startup keeps answering ready: true, phase: "ready" and the torrents now loaded, per engine.

A probe that wants "ready" reads the status word of /health or ready of /api/startup. A liveness probe that should not kill a long start accepts starting.

The image has no HEALTHCHECK and no curl; this one uses bash. Match the whole status word: grep -q healthy also matches "unhealthy", so it would pass a 503.

    healthcheck:
      test:
        - CMD
        - bash
        - -c
        - exec 3<>/dev/tcp/127.0.0.1/8199 && printf 'GET /health HTTP/1.0\r\n\r\n' >&3 && grep -qE '"status":"(healthy|degraded|starting)"' <&3
      interval: 60s
      timeout: 10s
      retries: 3

It passes on starting, healthy and degraded, and fails on unhealthy, rescue mode (ok) and no answer. Drop degraded to flag a missing listener, a kill-switch block or a start_paused hold. Docker only marks the container unhealthy; it does not restart it. See also Install with Docker.

Quick check from a shell:

curl -s http://localhost:8199/health | jq '{status, problems}'

GET /metrics needs no API key and returns Prometheus text (text/plain; version=0.0.4, with # HELP and # TYPE): 32 families, per engine (extra engines included): rates, bytes, peers, torrents per state, announces (per tracker host too), speed caps, choker, listener and start_paused state, pieces in flight, and jemalloc memory (Linux/Unix). A scrape never walks the catalogue. While Hydranos starts it serves a reduced set (hydra_up, hydra_uptime_seconds, hydra_starting, hydra_startup_restored, hydra_startup_records).

The full metric table, a scrape config and ready-made alert rules are on Prometheus Metrics.

The page is public: anyone who can reach port 8199 can read it, including the tracker host names (tracker label) and your volumes. See Security and Access.

Everything the Overview, Benchmark and Trackers tabs show comes from JSON routes that need the API key. Their fields are listed under Monitoring on REST API:

Without Prometheus, a small cron check is enough. This one exits non-zero when something needs a look; plug it into whatever notifies you (cron mail, a healthchecks service, your own script):

#!/usr/bin/env bash
# hydranos-check.sh: run every 5 minutes from cron
HY=http://localhost:8199
KEY=YOUR_API_KEY

h=$(curl -s --max-time 10 "$HY/health") || { echo "Hydranos does not answer"; exit 2; }
st=$(jq -r '.status' <<<"$h" 2>/dev/null)
case "$st" in
  healthy|starting) ;;
  degraded) jq -r '.problems[].detail' <<<"$h"; exit 1 ;;
  *) echo "health: ${st:-not JSON (rescue mode?)}"; jq -r '.problems[]?.detail' <<<"$h" 2>/dev/null; exit 2 ;;
esac

red=$(curl -sf --max-time 60 -H "X-Api-Key: $KEY" "$HY/api/announce/health" | jq '.badges.trackers_red')
[ "${red:-0}" -gt 0 ] && { echo "$red tracker(s) red: passkey refused or missing from a peer list"; exit 1; }
exit 0

/api/announce/health walks every tracker host of every engine from the in-memory announce caches; its cost grows with the number of trackers, not of torrents. On Windows, run the same requests with curl.exe from a scheduled task, or Invoke-RestMethod in PowerShell.

Every log event goes to three places at once, all filtered by the same level:

If the file cannot be opened, Hydranos prints hydranos: no log file at <path> (<error>); console only and keeps running.

Timestamps in /api/logs are UTC in RFC 3339 form, ending in Z. The Logs tab shows the same UTC time with the T and the Z removed and cut to milliseconds, so nothing on screen says it is UTC: it is not your local time.

Hydranos rotates hydranos.log itself, on every platform. When the next line would take the file past 128 MiB, it renames hydranos.log to hydranos.log.1 (the older files move up to .2, .3, .4; the previous .4 is deleted) and starts a new hydranos.log. At most five files exist, about 640 MiB in all. The rotated files are not compressed. The size count resumes from the existing file at startup, so a restart does not reset it.

There is nothing to configure and no logrotate rule is needed. Every incoming peer connection still writes an INFO line ([peer] incoming … from …), so on a busy instance the 640 MiB cover less time than you might expect; to keep more history, lower the level with RUST_LOG (below) or collect the console with your log system.

If HYDRANOS_LOG_STDOUT is set to any value (including 0, false or an empty string), no file is written and only the console and the ring remain. The console output is the same whether or not the variable is set.

Under Docker this is usually what you want: docker logs already keeps the console, capped by the compose logging: block. Without a console (a Windows double-click start, for example) the file is your only record of a failed start, so keep it there.

RUST_LOG sets the level for all three outputs. When it is unset or cannot be parsed, the level is info. It uses the standard tracing filter syntax: a default level, then target=level pairs.

Docker: add RUST_LOG: "info,typhon_engine::peer=warn" under environment:. systemd: Environment=RUST_LOG=info,typhon_engine::peer=warn in an override (systemctl edit hydranos). Where each variable is parsed: Configuration: Environment Variables, CLI and Signals.

There is no HTTP access log: requests to the API are not logged.

The Logs tab shows the ring buffer: the last 2000 lines since the process started, filtered by the server. Each line shows time (UTC, see above), level, module (the code area, without the hydranos:: prefix, for example announce::runner) and message. Structured fields printed in the console and the file (for example torrents= or addr=) are appended to the message as key=value.

2000 lines can cover only minutes on a busy instance: for older lines read hydranos.log (and its rotated files) or the console.

The same data over the API (API key required):

curl -s -H "X-Api-Key: YOUR_API_KEY" "http://localhost:8199/api/logs?level=warn&since=1h" \
  | jq -r '.entries[] | "\(.ts) \(.level) \(.module) \(.msg)"'

GET /api/logs returns {"entries":[{"ts","source","level","module","msg","seq"}, …],"last_seq":N,"modules":[…]}, oldest first; source is always rust. Optional parameters: level (minimum: trace, debug, info, warn, error), module (or source; substring, source=rust ignored), q (or contains; case-insensitive text of the message and fields), since (90s, 5m, 1h, 24h, 2d or Unix seconds), after (a seq) and limit (the newest N matches). A value it cannot read (since=1week, level=loud) answers 400 {"error":…} instead of being silently dropped.

GET /api/logs/stream is a server-sent event stream (API key in the header or ?apikey=) with events named log, each carrying one entry and its seq as the event id. It accepts the same filters except since, starts after after= (or the browser's Last-Event-ID on a reconnect, otherwise from now), and ends by itself when Hydranos stops.

The Race engine / Hoard engine cards (except the Race card's Session ratio, computed over all engines' session bytes) show the engines with the ids race and hoard. Every other engine (for example vpn1) has its own card on the Overview, its own live cards and an upload curve on the Benchmark tab, and counts in the Records (except Peak live seeds), the header totals, the Storage card, tracker health and /metrics.

What still covers only race and hoard: the hoard-only announce cards, Peak live seeds, the A / B comparison (below), and the race timeline and its snapshots (engine race only). See Engines: Race and Hoard.

• All-time upload: the UL total figure in PiB, the lifetime ratio, total downloaded, the torrent count (all engines), and progress to the next whole PiB. Its milestone rows come from bench.db.
• Race engine / Hoard engine cards: Torrents, With peers, Upload, Download, Peers, Seeding (that engine's seed size) and Session ratio (since the process started; on the Race card, all engines' session upload ÷ all engines' session download).
• Engine {id} cards, one per extra engine (refreshed at most every 3 s): the tag is the engine's role, or <role> · not listening; rows Torrents, With peers, Upload, Download, Peers, Seeding (- before the first tracker pass) and Session ratio (that engine's own).
• Storage: compares what trackers credit with the space it takes.

The link scanner waits about 3 minutes after startup, then walks the catalogue in batches. A first count of a large library takes a long time; until it lands, the rows show -.

• Records: all-time peaks computed from bench.db: Peak upload, Peak download, Peak swarm peers (all engines), Best upload day, Peak live seeds (race + hoard only) and Best line test (the best line-test upload, in Gbps). The server recomputes them at most every 30 minutes. With nothing recorded yet the card says No records yet.

The Benchmark tab reads bench.db in the data dir. One sample row is written every 5 s (with one more row per extra engine, in its own table); per-tracker rows every 30 s. Tracker rows older than 48 hours are folded into 5-minute buckets. The sample tables are never pruned, so bench.db grows by about 17,000 sample rows a day (plus as many per extra engine) for as long as it exists. If bench.db cannot be opened, Hydranos logs no bench database: … and the tab stays empty.

• Live cards: Upload Race, Download Race, Upload Hoard, Peers Race (connected peers), IOWait, ARC Hit Rate, ARC Miss/s, ARC Size; then, per extra engine, Upload {id}, Download {id} and Peers {id} (the row is hidden when there is none).
• Announce cards for the hoard engine only: Announces (per second over 5 minutes, against what the intervals require), Late (deadline passed by more than 5 s), In flight, Refused (429) (share of 429 answers over 10 s; above 2 % the announcer backs off). How the announcer works: Trackers and Announces. For other engines use hydra_announce_* on Prometheus Metrics.
• Charts over 15m, 1h, 6h, 24h, 7d or 30d: Upload Speed (one curve per extra engine too), Total Uploaded (extras stacked as Other engines), Currently Uploading, Announce Rate, Race Events, IOWait %, ARC Miss / sec.

IOWait needs Linux /proc; the ARC figures need ZFS. Without them they read 0: that means "not measured", not "no disk pressure". On Windows none of them is measured.

▶ Run now (POST /api/vpn-speedtest/run) runs iperf3 against [vpn_speedtest] iperf3_server: upload first, then download (-R), never both at once, duration_secs each. The button shows ⏳ Running… and the request waits for the end (about twice duration_secs). The test goes out the same way as the engines: through the bind_interface of the engine whose role is race (or the first engine), a managed WireGuard tunnel included, with --bind-dev (Linux); with no interface, through the default route. The torrents keep running; their rate is read halfway through each direction and shown beside the result.

The chart Upload / Download (Mbps), last 7 days shows the history, kept in bench.db (the Best line test record comes from it). Errors: 400 iperf3_server not configured in [vpn_speedtest] (also when enabled = false) or the engines go out through a SOCKS5 proxy, which iperf3 cannot use; 409 a speedtest is already running; 502 for an iperf3 failure, named per direction: upload: … and/or download: … joined by , (for example upload: cannot run iperf3: … or download: iperf3 did not finish in 40s, the limit being twice the test duration + 20 s). Only one engine's way out is measured.

iperf3 ships in the Docker image; elsewhere it must be on the PATH. Each test saturates the line for about twice duration_secs.

[vpn_speedtest]
enabled = true                      # absent = false
iperf3_server = "ping.online.net"   # absent = "" (no test)
# iperf3_port = 5201                # absent = 5201
# duration_secs = 10                # seconds per direction, 1-60; absent = 10
# interval_secs = 0                 # automatic test every N s (never under 300); 0 or absent = Run now only

The keys are read at each run, so a change made in Config applies to the next test without a restart. With interval_secs above 0, the first automatic test runs one interval after the start, and a failure logs periodic speedtest: ….

Upgrading: a config file started from the 4.3 default.toml (or carried over from 3.x) may contain interval_secs = 3600. In 4.3 the key was ignored; since 4.4 it is read, so such a file runs a line test every hour after the upgrade. Set interval_secs = 0 (or remove it) to keep Run now only.

A / B comparison compares two contiguous periods around a change: fill Start of period 1, Middle point (change), End of period 2, then Compare. The table lists, per sampled metric, P1 avg, P1 max, P2 avg, P2 max and Δ avg (relative to P1, green or red depending on which direction is better), with P1: {p1} samples, P2: {p2} samples. Metrics without a friendly name show their column name (for example global_uploaded, open_fds). No data for this range. means one side has no sample.

GET /api/benchmark/compare?start=&mid=&end= (Unix seconds, start < mid < end) returns the same with avg, max, p95 and count per side and delta_avg_pct; the p95 is not shown in the UI.

Known bug in 4.4: race_avg_share and hoard_session_uploaded are in the comparison but the sampler never writes them, so they always read 0 and Δ 0 % — read the other rows. Extra engines have no row in the comparison: compare their curves on the Upload Speed chart, or their /metrics series.

Every 5 s, for each torrent of the engine with the id race that is not complete yet, Hydranos records a snapshot with its ten fastest peers. The race timeline draws them; GET /api/benchmark/race-snapshots/:info_hash returns them alone (null when there are none). They are deleted with the torrent's race events when it leaves the engine. Extra engines with the race role have no snapshots.

Accepted but ignored in 4.4: the [bench] section (enabled, retention_days, race_snapshot_retention_days, snapshot_interval_secs, prune_interval_mins, vacuum) looks like recording and retention settings — the startup log warns config: [bench] is ignored: bench recording and its retention are built in; none of these keys is read. Instead: nothing; delete the section. — to reclaim space, stop Hydranos and delete or archive bench.db; it is recreated empty at the next start and you lose the history, line tests and Records.

The Trackers tab shows, per tracker host, the announce errors of the last 60 minutes by class and a red/amber severity; the tab badge counts red and amber trackers. The full description is on Trackers and Announces.

For scripts, the same data is available with the API key:

curl -s -H "X-Api-Key: YOUR_API_KEY" http://localhost:8199/api/announce/health
curl -s -H "X-Api-Key: YOUR_API_KEY" "http://localhost:8199/api/announce/errors?host=tracker.example.org"

Both cover every engine; a tracker failing on two engines counts once in the badge.

GET /api/health/anomalies (API key required; also the MCP health tool) serves the last result of a background pass plus live counters: the call itself never walks the catalogue. The pass runs 2 minutes after the start, then every 5 minutes, over every engine; when it finds something it logs health scan found something (anomalies=). The web UI does not call this endpoint.

Other fields: scan (pending until the first pass, then done), generated_at, scan_duration_ms, scanned (per engine, plus scanned_race / scanned_hoard), anomalies (up to 500 entries {type, engine, info_hash, name, detail, wasted_bytes}, anomalies_truncated beyond), wasted_bytes, efficiency (useful ÷ downloaded bytes), redl_historical / redl_historical_bytes (over 1.2 × size and 50 MiB more), trackers ({host, severity, engines}), trackers_red, announces (late, held_engines), disks and engines (torrents and states per engine).

disks has one line per filesystem holding data_dir or a save path seen at the last pass: path, free_bytes, total_bytes, free_pct, and low when less than 5 GiB or less than 2 % is free.

On Linux and other Unix builds, Hydranos uses the jemalloc allocator and reacts to two signals. Neither exists on Windows.

docker kill --signal=SIGUSR1 hydranos       # Docker
sudo systemctl kill -s SIGUSR2 hydranos     # systemd

The jemalloc allocated=… resident=… line every 5 minutes, and hydra_memory_allocated_bytes / hydra_memory_resident_bytes on /metrics, tell you which case you are in: if resident follows allocated, the memory is really in use; if resident is much larger, SIGUSR2 will give most of it back. The Docker image pays a small sampling overhead for profiling all the time; to turn it off, set MALLOC_CONF to an empty value in your compose environment:.

• Operating Hydranos: stop, restart, rescue mode, backups, updates.
• Trackers and Announces: the Trackers tab in full.
• Troubleshooting: symptoms and fixes.
• Configuration: Environment Variables, CLI and Signals: RUST_LOG, HYDRANOS_LOG_STDOUT, signals.
• Install with Docker: health check and log cap in compose.
• Prometheus Metrics: every /metrics family, scrape config, alert rules.