Metrics with Prometheus - ptweezy/cronstable GitHub Wiki

Metrics with Prometheus

cronstable exposes Prometheus metrics at a built-in GET /metrics endpoint on the HTTP web API. This is the pull-side sibling of the per-job statsd push metrics. On demand, a Prometheus server scrapes cumulative counters (run outcomes, durations, retries), live gauges (running instances, next scheduled run, last-run outcome), and the full cluster health surface. The exposition is generated by cronstable itself: no exporter sidecar and no extra dependency, so it works on every architecture and platform cronstable ships for, including Windows.

The endpoint serves both exposition formats: the classic Prometheus text format (text/plain; version=0.0.4) by default, and OpenMetrics 1.0 (application/openmetrics-text) when the scraper advertises it in its Accept header, as modern Prometheus servers do.

Enabling the endpoint

The endpoint is on by default whenever the web API is enabled. A web.listen block is enough:

web:
  listen:
    - http://127.0.0.1:8080

$ curl http://127.0.0.1:8080/metrics
# HELP cronstable_info cronstable build information.
# TYPE cronstable_info gauge
cronstable_info{version="1.2.60"} 1
...

The web.metrics option tunes or disables it. It accepts a boolean shorthand or a map:

web:
  listen:
    - http://127.0.0.1:8080
  # metrics: false            # shorthand: disable the endpoint entirely
  metrics:
    enabled: true             # default
    public: false             # default; see Authentication below
    durationBuckets:          # histogram bounds override (seconds)
      - 1
      - 10
      - 60
      - 600

Options

The schema for the block is Bool() | Map({Opt("enabled"): Bool(), Opt("public"): Bool(), Opt("durationBuckets"): Seq(Float())}). Every key is optional.

Option Type Default Description
enabled bool true Serve GET /metrics. metrics: false / metrics: true are shorthands for {enabled: ...}.
public bool false Exempt /metrics (and only /metrics) from web.authToken bearer-token authentication, for scrapers that cannot send credentials.
durationBuckets list of float 0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600 Upper bounds (seconds) of the cronstable_job_duration_seconds histogram. Must be finite, positive, and strictly increasing. A +Inf bucket is always appended automatically. Changing the bounds on a reload restarts the histograms from zero (an ordinary counter reset to Prometheus).

Scraping and authentication

When web.authToken is configured, /metrics requires the bearer token like every other data endpoint. Prometheus supports this directly in its scrape configuration:

scrape_configs:
  - job_name: cronstable
    static_configs:
      - targets: ["cron-host:8080"]
    authorization:
      type: Bearer
      credentials: "the-secret-token"

If your scraper cannot send credentials, set web.metrics.public: true to exempt /metrics from authentication while everything else stays gated. The metrics reveal job names, schedules, and outcomes, but never command lines, output, or secrets. Treat public accordingly.

Two response details:

  • web.headers are stamped onto the response as usual, except Content-Type: the exposition format's contract always wins.
  • Content negotiation is on the Accept header. Any request advertising application/openmetrics-text gets OpenMetrics, which ends with # EOF. Everything else gets the text format.

Metric reference

All metrics are prefixed cronstable_. Per-job metrics carry the job name in a job_name label, not job, which would collide with Prometheus's own target label. Counters are cumulative since process start. They reset when the daemon restarts, which Prometheus's rate()/increase() handle as built-in behavior. With a durable state store configured, the per-job counters are re-seeded across restarts; see semantics and guarantees.

Daemon

Metric Type Description
cronstable_info{version} gauge (info) Build information; always 1.
cronstable_start_time_seconds gauge Unix time this cronstable process started.
cronstable_job_set_info{job_set_id} gauge (info) The job-set fingerprint of the currently loaded configuration; always 1. Compare across a fleet to spot configuration drift.
cronstable_jobs{state} gauge Number of configured jobs, state="enabled" / "disabled".
cronstable_config_last_reload_successful gauge 1 if the most recent configuration parse succeeded, 0 if the configuration on disk is currently broken (the daemon keeps running the previous good one). Absent until the first parse.
cronstable_config_last_reload_success_timestamp_seconds gauge Unix time of the last successful configuration parse.

Per job

Metric Type Description
cronstable_job_runs_total{job_name, status} counter Finished runs by outcome: status="success" / "failure" / "cancelled" / "skipped", exactly as recorded in the run history. skipped counts the synthetic ledger rows a paused job writes in place of a launch. Those rows stamp neither last-outcome timestamp. Zero-filled for every configured job from the first scrape. Runs replaced under concurrencyPolicy: Replace are not runs and are not counted, matching the dashboard.
cronstable_job_duration_seconds histogram Duration of finished runs (bucket bounds from durationBuckets). Cancelled-without-starting records carry no duration and are not observed.
cronstable_job_retries_total{job_name} counter Retry attempts actually launched (onFailure.retry). Deferred or abandoned retries are not counted.
cronstable_job_permanent_failures_total{job_name} counter Failures with no retry remaining: exactly when onPermanentFailure reporting fires.
cronstable_job_start_failures_total{job_name} counter Runs whose command could not be launched at all, also recorded as failures with the conventional exit code 127.
cronstable_job_last_success_timestamp_seconds{job_name} gauge Unix time of the last successful finish. Absent until the first success.
cronstable_job_last_failure_timestamp_seconds{job_name} gauge Unix time of the last failed finish. Absent until the first failure.
cronstable_job_info{job_name, schedule, cluster_policy} gauge (info) Static configuration facts; always 1.
cronstable_job_enabled{job_name} gauge 1 if the job is enabled in the loaded configuration.
cronstable_job_running{job_name} gauge Number of currently running instances (can exceed 1 under concurrencyPolicy: Allow).
cronstable_job_next_run_timestamp_seconds{job_name} gauge Unix time of the next scheduled run. Absent for disabled jobs and @reboot schedules.
cronstable_job_last_run_timestamp_seconds{job_name} gauge Unix time the most recent run finished.
cronstable_job_last_run_duration_seconds{job_name} gauge Duration of the most recent finished run.
cronstable_job_last_run_exit_code{job_name} gauge Exit code of the most recent finished run.
cronstable_job_last_run_success{job_name} gauge 1 if the most recent finished run succeeded, else 0 (a cancelled run counts as 0).
cronstable_job_cpu_seconds_total{job_name, mode} counter CPU time consumed by monitorResources runs, mode="user" / "system", summed over the job's monitored runs. Emitted only after the job has a monitored run (an unmonitored job exports no CPU series).
cronstable_job_peak_rss_bytes{job_name} gauge Highest resident-set size (bytes) observed across all the job's monitored runs. Absent until the first monitored run.
cronstable_job_last_run_cpu_seconds{job_name} gauge Total CPU time of the most recent monitored run. Absent unless that run had monitorResources on.
cronstable_job_last_run_max_rss_bytes{job_name} gauge Peak resident-set size (bytes) of the most recent monitored run. Absent unless that run had monitorResources on.
cronstable_job_paused{job_name} gauge 1 while the job is paused at runtime, else 0.
cronstable_job_late{job_name, check} gauge 1 while the job's SLA check is breached, per check (maxTimeSinceSuccess / lateAfter / maxRuntime). Emitted from the monitor's first evaluation, only for checks the job configures in its sla: block. Paused jobs are exempt.
cronstable_job_sla_breaches_total{job_name, check} counter SLA breaches latched for the job, per check. Increments once per breach, not per evaluation. Emitted alongside cronstable_job_late, zero-filled from the first evaluation.

The four preceding resource families require the job to opt into monitorResources. They are sampled and best-effort: see resource monitoring for the sampling model and every other surface the numbers reach. Like the run counters, cronstable_job_cpu_seconds_total and cronstable_job_peak_rss_bytes are made restart-durable by a state: store (see semantics and guarantees).

State backend

Emitted when a durable state store is configured (a state: section). Without one, cronstable is stateless and these families are absent. They observe the store itself. The preceding job families are unchanged, except that the store makes the per-job counters restart-durable (see semantics and guarantees). Families and label values appear as they first become relevant, so a store that never contends a lock or is never throttled does not export frozen zero series forever.

Metric Type Description
cronstable_state_info{backend, topology} gauge (info) Static store facts: the backend implementation (filesystem) and the effective topology (single-node / shared, after topology: auto resolves); always 1.
cronstable_state_ops_total{op} counter Store operations by kind, for example op="append" / "list" / "derive-max" / "prune" / "lease-acquire" / "lease-renew" / "lease-release" / "lease-read" / "start" / "gc". Each op value appears after that operation has first run.
cronstable_state_op_errors_total{op} counter Store operations that raised, same op label.
cronstable_state_op_seconds_total{op} counter Seconds spent inside store operations. For mean in-store latency per operation, divide by cronstable_state_ops_total, for example rate() over rate().
cronstable_state_lock_acquisitions_total counter Advisory-lock acquisitions by the store. Emitted after it goes nonzero.
cronstable_state_lock_wait_seconds_total counter Seconds spent waiting to acquire the store's advisory locks: the lock-contention signal. Emitted alongside cronstable_state_lock_acquisitions_total.
cronstable_state_throttled_ops_total counter Store operations delayed by the state.maxOpsPerSecond token bucket. Emitted after it goes nonzero.
cronstable_state_throttle_wait_seconds_total counter Seconds those operations spent queued behind the rate limit. Emitted alongside cronstable_state_throttled_ops_total.
cronstable_state_dropped_writes_total{kind} counter Durable writes that failed and were dropped (with a logged warning), by kind="run-record" / "checkpoint" / "retry" / "reboot-marker" / "counters" / "manifest" / "inflight" / "pause" / "overflow" (a write shed because 8192 were already pending). Writes are fire-and-forget under either onStoreUnavailable policy. Emitted after it goes nonzero: the single most alertable state signal.
cronstable_state_workers_inflight{lane} gauge State worker threads in flight per lane, lane="bulk" / "lease". Sustained at capacity signals a slow or wedged mount.
cronstable_state_workers_capacity{lane} gauge Worker-thread concurrency cap per lane.
cronstable_state_workers_peak{lane} gauge Highest number of worker threads in flight per lane.

If reading the store's stats fails during a scrape, the state families are omitted from that scrape, with the error logged, rather than the scrape failing. The following cluster block follows the same degradation contract. cronstable_state_dropped_writes_total is the exception: the scheduler accumulates it as it drops writes, and it is not read from the store, so it keeps being served even when the store itself cannot be read.

Cluster

cronstable_cluster_enabled is always present (0 with no cluster section or a failed backend). The rest of the block appears while a cluster backend runs. The values mirror the pre-derived fields of GET /cluster, so the same alert rules documented there apply.

Metric Type Description
cronstable_cluster_enabled gauge 1 when a leadership backend is running on this node (0 also covers a configured backend that failed to start).
cronstable_cluster_info{backend, node_name, distribution} gauge (info) Backend (gossip/kubernetes/etcd/filesystem), this node's name, and the ownership distribution; always 1.
cronstable_cluster_size / cronstable_cluster_quorum gauge Effective cluster size N and the quorum threshold (lease backends report 1/1).
cronstable_cluster_quorate gauge Whether this node is part of a quorum: the single most alertable cluster signal.
cronstable_cluster_is_leader gauge Whether this node holds scheduled-job leadership. Always 0 under distribution: spread (ownership is per job there), mirroring GET /cluster.
cronstable_cluster_leader_info{leader} gauge (info) The observed leader's name; absent when there is none.
cronstable_cluster_conflict{kind} gauge The three fail-closed conflict gates, kind="nodename" / "size" / "policy". Any 1 means this node's Leader jobs are standing down.
cronstable_cluster_peers{status} gauge Gossip backend only: configured peers by observed status (agreed, syncing, drifted, unreachable, untrusted, conflict, self, unknown), zero-filled so alert series never vanish.
cronstable_cluster_leader_transitions_total counter Times this node acquired or lost leadership. Observed at scheduler cadence (once per minute), so a flap shorter than a scheduler tick may be missed. Emitted only when electLeader is on (an observe-only cluster has no leadership to transition).
cronstable_cluster_quorum_transitions_total counter Times this node joined or left quorum (same cadence and electLeader caveats).

If a backend read fails during a scrape, the cluster block degrades to cronstable_cluster_enabled and the transition counters (with the error logged by the prometheus logger) rather than the whole scrape failing.

Semantics and guarantees

  • Metrics never affect scheduling. Every hook is a synchronous in-memory increment on the scheduler's event loop. There is nothing to time out, retry, or crash on. This is the same contract the statsd writer has, without the UDP.

  • Counters agree with the API. cronstable_job_runs_total increments at the exact point a run enters the run history (GET /jobs/{name}/runs), so the two surfaces can never disagree on outcomes.

  • Counters survive reloads. With durable state, they survive restarts. The accumulators live on the daemon, not the web app, so even a configuration reload that restarts the web server or the cluster manager keeps them. Without a state: section, a process restart resets them, which is normal for Prometheus counters. With a durable state store configured, the per-job counters are snapshotted durably and seeded back on boot for the jobs still in the configuration:

    • Counters covered: runs by outcome, retries, permanent and start failures, the duration histogram, the last success/failure timestamps, and the SLA breach counters.
    • Cadence: at most one write per 15 seconds while running, plus a final snapshot on clean shutdown.
    • Histograms: seeded only if the durationBuckets bounds are unchanged.

    A restart then reads as, at worst, a small ordinary counter reset covering the events since the last snapshot, and a crash forfeits at most those last few seconds. cronstable_start_time_seconds still resets per process by design: that is how you see the restart itself.

  • Removed jobs disappear. On every successful reload, the daemon drops series for jobs no longer in the configuration. A renamed job starts over at zero.

  • @reboot keep-alives look "always running". A long-running @reboot job shows cronstable_job_running 1 with zero finished runs. That is the correct reading, not missing data.

Example alerts

groups:
  - name: cronstable
    rules:
      # a job's most recent run failed
      - alert: CronstableJobFailed
        expr: cronstable_job_last_run_success == 0
        for: 5m
      # a job that should run hourly has not succeeded for 2 hours
      - alert: CronstableJobStale
        expr: time() - cronstable_job_last_success_timestamp_seconds > 7200
      # the config file on disk is broken; the daemon is running stale config
      - alert: CronstableConfigBroken
        expr: cronstable_config_last_reload_successful == 0
        for: 10m
      # this node has lost quorum (gossip) or cannot reach the lease store;
      # its Leader jobs are standing down
      - alert: CronstableClusterNotQuorate
        expr: cronstable_cluster_quorate == 0
        for: 5m
      # more than one node believes it is the leader (split-brain)
      # (inert under distribution: spread; compare clusterOwner instead)
      - alert: CronstableMultipleLeaders
        expr: sum(cronstable_cluster_is_leader) > 1

The clustering monitoring guide discusses the cluster signals in depth.

The preceding staleness rule (CronstableJobStale) is the external backstop to pair with per-job late-run detection. The in-process sla: monitor gives per-job thresholds and rich onLate notifications, but it stops with the daemon. A Prometheus rule on time() - cronstable_job_last_success_timestamp_seconds, plus up == 0 on the scrape itself, still fires when the daemon, its host, or its network is gone.

See also