Metrics with Statsd - ptweezy/cronstable GitHub Wiki

Metrics with statsd

cronstable sends per-job lifecycle metrics to a statsd server over UDP. This page covers configuration, wire format, and delivery behavior. You can enable statsd alongside the built-in Prometheus endpoint.

Enabling statsd for a job

Add a statsd block to a job or a defaults block. Jobs without it send no statsd metrics.

jobs:
  - name: test01
    command: echo "hello"
    schedule: "* * * * *"
    statsd:
      host: my-statsd.example.com
      port: 8125
      prefix: my.cron.jobs.prefix.test01

Options

All three options are required when statsd is configured:

Option Type Default Description
host string (required when statsd set) Hostname or IP of the statsd server. Resolved when cronstable opens its pooled UDP endpoint. The endpoint is rebuilt at least every 60 seconds, or after a UDP error, so a DNS change takes effect within a minute. An unresolvable host produces a warning, not a crash (see best-effort delivery).
port int (required when statsd set) UDP port of the statsd server (commonly 8125).
prefix string (required when statsd set) Metric name prefix. cronstable joins it to each suffix with a . (<prefix>.start), so omit a trailing dot. CR, LF, :, and | are removed from the prefix.

Wire format

Metrics are encoded as UTF-8 statsd lines, one metric per line, ending with a newline (\n). They are sent in two UDP datagrams per job run: one at start, one at stop.

On start

After the subprocess launches, cronstable records time.perf_counter() as the start time and sends a single datagram:

<prefix>.start:1|g

|g is the statsd gauge type. The value is always 1.

A command that fails to launch, such as a missing executable, emits neither start nor stop metrics.

On stop

When the job stops (normal exit, timeout, or cancellation), cronstable computes the duration as time.perf_counter() - start_time, converts it to milliseconds with int(round(duration_seconds * 1000)), and sends a single datagram containing three metrics:

<prefix>.stop:1|g
<prefix>.success:<1|0>|g
<prefix>.duration:<ms>|ms
  • <prefix>.stop:1|g: gauge, always 1.
  • <prefix>.success:<1|0>|g: gauge. 1 if the job did not fail, 0 if it failed. The value comes from 0 if job.failed else 1, where job.failed is the failure-detection result (failsWhen). A nonzero exit code, output on a watched stream, or failsWhen: always therefore produces success:0.
  • <prefix>.duration:<ms>|ms: timer (|ms) with no sample-rate suffix. The numeric value is the integer wall-clock duration in milliseconds, measured with perf_counter between start and stop. A run sends at most one duration observation.

When the job has monitorResources on (see resource monitoring), the same stop datagram also carries the run's resource accounting:

<prefix>.cpu:<ms>|ms
<prefix>.max_rss:<bytes>|g
  • <prefix>.cpu:<ms>|ms: timer, the total CPU time (user + system) of the run's process tree in milliseconds. Like duration, it is sent once per run with no sample-rate suffix.
  • <prefix>.max_rss:<bytes>|g: gauge, the peak resident-set size observed during the run, in bytes.

These two lines are omitted for a run that was not monitored (or where sampling caught nothing), so an unmonitored job's stop datagram is unchanged.

A run killed at its resourceLimits.memory cap adds one more line to the same stop datagram:

<prefix>.memory_limit_exceeded:1|c

|c is the statsd counter type. Every other run, capped or not, sends no such line.

A run whose command never launches (the subprocess could not be spawned, or resourceLimits.required refused the run before the spawn; start_failed is set) emits neither metric. On that path, wait() returns early, _on_stop (hence job_stopped) is never called, and _on_start was likewise never reached. Separately, job_stopped itself guards on a recorded start time (if self.start_time is None: return). That guard suppresses a stop metric for a run stopped without a corresponding job_started (for example, the cancel()/wait() race under concurrencyPolicy: Replace).

Best-effort delivery

statsd metrics are telemetry and never affect job execution or the scheduler.

  • Fire-and-forget UDP. cronstable keeps one UDP endpoint per statsd host and port and reuses it for later sends, reopening it every 60 seconds or after an error. Nothing is acknowledged or retried. If more than 1 MiB is queued on an endpoint, new datagrams are dropped. The network or OS silently drops lost datagrams. Inbound datagrams are ignored.
  • Send failures are caught and logged, never fatal. _on_start catches OSError; _on_stop catches any Exception. A failure, such as an unresolvable host, is logged with logger.warning(...) plus exc_info=True, and the job proceeds normally. The messages are:
    • Job <name>: failed to send statsd job_started metric
    • Job <name>: failed to send statsd job_stopped metric
  • UDP protocol errors are logged separately. The statsd logger records asyncio-level UDP errors surfaced through the datagram protocol as UDP error received: <exc> (see logging configuration for logger names).

Stop metrics are emitted at most once per run

The stop metrics are guaranteed to be sent at most once per run, even when two code paths could both reach the stop logic. _on_stop is idempotent: it checks a _stopped flag and returns early if already stopped, setting the flag before any await.

async def _on_stop(self) -> None:
    if self._stopped:
        return
    self._stopped = True
    ...

With concurrencyPolicy: Replace, cancellation and completion may both call _on_stop; only the first call emits metrics. See concurrency and timeouts for the policies and failure detection and retries for how success is computed.

A run canceled during its command gets _on_stop at the end of cancel(). A cancel that arrives while the run's verification check runs terminates the check and does not call _on_stop; wait() calls it once it has collected the check. The stop datagram for that run therefore goes out when the daemon has collected the check, success reflects the check's outcome, and duration runs to that moment.

The start_time guard means a forced cancellation also yields a correct duration. The duration is measured from the recorded perf_counter start to the moment job_stopped runs, regardless of how the process ended (normal exit, executionTimeout, or Replace cancellation).

See also

⚠️ **GitHub.com Fallback** ⚠️