Prometheus Metrics - Kheopsian/Hydranos GitHub Wiki

πŸ‡¬πŸ‡§ English Β· πŸ‡«πŸ‡· FranΓ§ais

Prometheus Metrics

Applies to Hydranos 4.4.

This page is the reference for GET /metrics: every family, its labels, what is served while Hydranos starts, a scrape config and alert rules. Health endpoints, logs and the other monitoring surfaces are on Monitoring and Logs.

GET /metrics needs no API key and returns Content-Type: text/plain; version=0.0.4. Every family has its # HELP and # TYPE lines. The figures come from counters Hydranos already keeps (the once-a-second rate pass, the announcer's counters, the last 30 s tracker pass): a scrape never walks the catalogue, so its cost does not grow with the number of torrents.

Per-engine families have one sample per local engine, extra engines included (engine="vpn1"). A family with no sample at all is not printed (for example hydra_seed_size_bytes before the first tracker pass, or hydra_tracker_errors_last_hour when no announce failed in the last hour).

Metrics

Metric Type Labels Meaning
hydra_up gauge β€” 1 while the process answers.
hydra_build_info gauge version Always 1; the running version is the label.
hydra_uptime_seconds gauge β€” Seconds since the process started, catalogue load included.
hydra_torrents gauge engine Torrents loaded in the engine.
hydra_torrents_by_state gauge engine, state Torrents per state: stopped, checking, downloading, seeding, error (error = the data is missing). There is no queued state: a torrent the download queue holds back (active_downloads) or a seed it parks (active_seeds) is counted under stopped.
hydra_torrents_paused gauge engine Torrents with their pause flag set: stopped by the operator, by a share limit, or held back by the download queue; also counted under their state above.
hydra_upload_rate_bytes gauge engine Upload rate, bytes/s.
hydra_download_rate_bytes gauge engine Download rate, bytes/s.
hydra_session_uploaded_bytes_total counter engine Bytes uploaded since the process started.
hydra_session_downloaded_bytes_total counter engine Bytes downloaded since the process started.
hydra_lifetime_uploaded_bytes gauge engine Lifetime upload of the torrents loaded now; falls when one is removed.
hydra_lifetime_downloaded_bytes gauge engine Lifetime download of the torrents loaded now; falls when one is removed.
hydra_peers gauge engine Connected peers.
hydra_torrents_with_peers gauge engine Torrents with at least one peer.
hydra_torrents_uploading gauge engine Torrents uploading now.
hydra_seed_size_bytes gauge engine Size of the seeding torrents as trackers credit it (a cross-seed counts once per tracker). Absent until the first tracker pass (every 30 s).
hydra_announces_total counter engine, result (ok, failed) Announces since the process started.
hydra_tracker_announces_total counter engine, tracker, result (ok, failed) Announces per tracker host since the process started.
hydra_tracker_errors_last_hour gauge engine, tracker, class Failed announces per tracker host and error class over the last 60 minutes. Absent when there is none. class is one of 9 values: rate_limited, timeout, invalid_passkey, unknown_torrent, dns, connect, http_error, udp_proxied, other (see Trackers and Announces).
hydra_announce_in_flight gauge engine Announces out now.
hydra_announce_slots gauge engine Announces allowed out at once (the scheduler's concurrency).
hydra_announce_late gauge engine Torrents whose announce deadline passed more than 5 s ago.
hydra_announce_needed_per_second gauge engine Announces per second the catalogue's intervals require.
hydra_rate_limit_bytes gauge engine, direction (up, down) Active speed cap in bytes/s, 0 = none; engine="client" is the client-wide cap above every engine: it is not an engine, exclude it from per-engine sums and panels ({engine!="client"}).
hydra_choking gauge engine 1 when the choker is on (choking = true).
hydra_unchoke_slots gauge engine Upload slots per torrent the choker grants (max_uploads_per_torrent; 0 or absent = 4). Published even while choking = false; absent when the slots are unlimited (negative value, the template's hoard).
hydra_listening gauge engine 1 when the engine's peer listener is bound.
hydra_held gauge engine 1 while start_paused holds the engine's dials and announces. An engine whose announces wait for a forwarded port (gluetun or managed WireGuard) is not counted here: see What /metrics does not show below.
hydra_pieces_in_flight gauge engine Pieces being fetched (reserved by a peer or a webseed, not yet verified).
hydra_piece_buffer_bytes gauge engine Bytes of received blocks held by the pieces in flight.
hydra_memory_allocated_bytes gauge β€” Bytes the process holds, as jemalloc counts them. Linux/Unix only.
hydra_memory_resident_bytes gauge β€” Resident bytes jemalloc maps; much above allocated means pages SIGUSR2 can return. Linux/Unix only.

Speed caps and the choker are described on Queue, Share and Speed Limits.

Series count. About 31 series per local engine (five of them for the states), 2 for engine="client" and 5 process-wide (hydra_up, hydra_build_info, hydra_uptime_seconds, the two memory gauges). Per engine and tracker host, add 2 for hydra_tracker_announces_total and up to 9 for hydra_tracker_errors_last_hour (only the classes that failed in the last hour). The number of torrents does not change it; the number of engines and tracker hosts does.

What /metrics does not show

Not exposed in 4.4: nothing about the network exit is in /metrics: no tunnel state or handshake age, no forwarded port, no kill switch verdict (covered, blocked, not covered), no exit IP, no DNS leak flag, and no sign that an engine's announces wait for a forwarded port (gluetun or managed WireGuard). Such an engine is not hydra_held, and its torrents do not become hydra_announce_late: each one is put back about a minute later, again and again, so its hydra_announces_total simply stops growing. The alert HydranosEngineSilent below catches that. For the rest, poll the JSON routes, which need the API key (X-Api-Key header) and so a small exporter of your own (or a JSON exporter):

Route What to read
GET /api/network/wireguard tunnels[], one per tunnel: engine, up, handshake_age_seconds, forwarded_port, degraded, dns_leak, last_error; engines[].assignment
GET /api/network/egress the kill switch: armed, each engine's verdict, blocked engines
GET /api/network/mode engine_state[]: per running engine, port_pending = true while its announces wait for a forwarded port
GET /api/port-forward per engine: port held, reachable, inbound_peers, forward (who forwards the port; external_port 0 while none is known)

/health reports an engine blocked by the kill switch or with no listener (degraded), not one waiting for its port. See Managed WireGuard and VPN and Egress.

No Grafana dashboard is shipped with 4.4. Starting points: sum by (engine) (hydra_upload_rate_bytes), sum by (engine, state) (hydra_torrents_by_state), hydra_announce_needed_per_second next to rate(hydra_announces_total[5m]), hydra_announce_late, sum by (tracker, class) (hydra_tracker_errors_last_hour), hydra_memory_resident_bytes.

While starting, /metrics already answers, with a reduced set: hydra_up, hydra_uptime_seconds, hydra_starting (1), hydra_startup_restored{engine} (resume records gone through so far) and hydra_startup_records{engine} (records to go through). The last three disappear once Hydranos is ready (there is no hydra_starting 0), and the per-engine families above appear.

Example (trimmed):

# HELP hydra_up 1 while the process answers.
# TYPE hydra_up gauge
hydra_up 1
# HELP hydra_torrents Torrents loaded in the engine.
# TYPE hydra_torrents gauge
hydra_torrents{engine="race"} 42
hydra_torrents{engine="hoard"} 15000
hydra_torrents{engine="vpn1"} 800
# HELP hydra_announces_total Announces since the process started.
# TYPE hydra_announces_total counter
hydra_announces_total{engine="hoard",result="ok"} 120394
hydra_announces_total{engine="hoard",result="failed"} 87

Scraping

Scrape config (prometheus.yml). No credentials are needed:

scrape_configs:
  - job_name: hydranos
    metrics_path: /metrics
    scrape_interval: 30s
    static_configs:
      - targets: ["hydranos.example.org:8199"]

If Prometheus runs in the same compose project, use the service name instead: targets: ["hydranos:8199"].

Several instances (for example a second one inside gluetun, declared as a node): scrape each one as its own target. Hydranos adds no instance or node label to its series; Prometheus adds instance (the target address), and you can add your own label per target. Each instance needs its own host port: publish the second one's 8199 on another host port ("18199:8199", on the gluetun container when it shares gluetun's network), or change its api_port (Configuration: Daemon, Auth and Global Sections).

    static_configs:
      - targets: ["hydranos.example.org:8199"]
        labels: { hydranos: "home" }
      - targets: ["hydranos.example.org:18199"]
        labels: { hydranos: "gluetun" }

Because the page is public, anyone who can reach port 8199 can read it β€” including, in the tracker label, the host names of every tracker you use, and your transfer volumes. Keep port 8199 off the internet or behind a reverse proxy that restricts /metrics; see Security and Access.

Alerts

groups:
  - name: hydranos
    rules:
      - alert: HydranosDown
        expr: up{job="hydranos"} == 0
        for: 5m
      - alert: HydranosRestartLoop
        expr: resets(hydra_uptime_seconds[1h]) >= 3
      - alert: HydranosTorrentsDropped
        expr: sum(hydra_torrents) < 0.9 * sum(hydra_torrents offset 1h)
        for: 15m
      - alert: HydranosEngineNotListening
        expr: hydra_listening == 0
        for: 5m
      - alert: HydranosEngineHeld
        expr: hydra_held == 1
        for: 30m
      - alert: HydranosAnnouncesFailing
        expr: |
          sum by (engine) (increase(hydra_announces_total{result="failed"}[15m]))
            / sum by (engine) (increase(hydra_announces_total[15m])) > 0.2
        for: 15m
      - alert: HydranosAnnouncesLate
        expr: hydra_announce_late > 1000
        for: 15m
      - alert: HydranosEngineSilent
        expr: sum by (engine) (increase(hydra_announces_total[30m])) == 0 and on (engine) hydra_torrents > 0
        for: 15m

hydra_uptime_seconds only rises during a process's life and restarts from 0, so resets() (documented for counters) counts restarts here: a single planned restart or upgrade does not fire HydranosRestartLoop; three in an hour do. HydranosTorrentsDropped catches a catalogue that shrank by more than 10 % (a bad restore, a drain or workflow that removed too much). HydranosEngineNotListening also fires for an engine you keep off the network on purpose (HYDRANOS_ENGINE_NET=0); exclude it with {engine!="…"}. The late and failure thresholds are examples: set them to what is normal for your library. hydra_tracker_errors_last_hour tells you which tracker and which error class is behind a failure alert. HydranosEngineSilent fires for an engine that announced nothing for 30 minutes while it holds torrents: announces waiting for a forwarded port, an engine held or blocked by the kill switch, or an engine whose torrents are all stopped.

See also