Metrics with Statsd - ptweezy/cronstable GitHub Wiki
cronstable sends per-job lifecycle metrics to a statsd server over UDP. This page covers configuration, wire format, and delivery behavior. You can enable statsd alongside the built-in Prometheus endpoint.
Add a statsd block to a job or a defaults block. Jobs without it send no statsd metrics.
jobs:
- name: test01
command: echo "hello"
schedule: "* * * * *"
statsd:
host: my-statsd.example.com
port: 8125
prefix: my.cron.jobs.prefix.test01All three options are required when statsd is configured:
| Option | Type | Default | Description |
|---|---|---|---|
host |
string | (required when statsd set) |
Hostname or IP of the statsd server. Resolved when cronstable opens its pooled UDP endpoint. The endpoint is rebuilt at least every 60 seconds, or after a UDP error, so a DNS change takes effect within a minute. An unresolvable host produces a warning, not a crash (see best-effort delivery). |
port |
int | (required when statsd set) |
UDP port of the statsd server (commonly 8125). |
prefix |
string | (required when statsd set) |
Metric name prefix. cronstable joins it to each suffix with a . (<prefix>.start), so omit a trailing dot. CR, LF, :, and | are removed from the prefix. |
Metrics are encoded as UTF-8 statsd lines, one metric per line, ending with a newline (\n). They are sent in two UDP datagrams per job run: one at start, one at stop.
After the subprocess launches, cronstable records time.perf_counter() as the start time and sends a single datagram:
<prefix>.start:1|g
|g is the statsd gauge type. The value is always 1.
A command that fails to launch, such as a missing executable, emits neither start nor stop metrics.
When the job stops (normal exit, timeout, or cancellation), cronstable computes the duration as time.perf_counter() - start_time, converts it to milliseconds with int(round(duration_seconds * 1000)), and sends a single datagram containing three metrics:
<prefix>.stop:1|g
<prefix>.success:<1|0>|g
<prefix>.duration:<ms>|ms
-
<prefix>.stop:1|g: gauge, always1. -
<prefix>.success:<1|0>|g: gauge.1if the job did not fail,0if it failed. The value comes from0 if job.failed else 1, wherejob.failedis the failure-detection result (failsWhen). A nonzero exit code, output on a watched stream, orfailsWhen: alwaystherefore producessuccess:0. -
<prefix>.duration:<ms>|ms: timer (|ms) with no sample-rate suffix. The numeric value is the integer wall-clock duration in milliseconds, measured withperf_counterbetween start and stop. A run sends at most one duration observation.
When the job has monitorResources on (see resource monitoring), the same stop datagram also carries the run's resource accounting:
<prefix>.cpu:<ms>|ms
<prefix>.max_rss:<bytes>|g
-
<prefix>.cpu:<ms>|ms: timer, the total CPU time (user + system) of the run's process tree in milliseconds. Likeduration, it is sent once per run with no sample-rate suffix. -
<prefix>.max_rss:<bytes>|g: gauge, the peak resident-set size observed during the run, in bytes.
These two lines are omitted for a run that was not monitored (or where sampling caught nothing), so an unmonitored job's stop datagram is unchanged.
A run killed at its resourceLimits.memory cap adds one more line to the same stop datagram:
<prefix>.memory_limit_exceeded:1|c
|c is the statsd counter type. Every other run, capped or not, sends no such line.
A run whose command never launches (the subprocess could not be spawned, or resourceLimits.required refused the run before the spawn; start_failed is set) emits neither metric. On that path, wait() returns early, _on_stop (hence job_stopped) is never called, and _on_start was likewise never reached. Separately, job_stopped itself guards on a recorded start time (if self.start_time is None: return). That guard suppresses a stop metric for a run stopped without a corresponding job_started (for example, the cancel()/wait() race under concurrencyPolicy: Replace).
statsd metrics are telemetry and never affect job execution or the scheduler.
- Fire-and-forget UDP. cronstable keeps one UDP endpoint per statsd host and port and reuses it for later sends, reopening it every 60 seconds or after an error. Nothing is acknowledged or retried. If more than 1 MiB is queued on an endpoint, new datagrams are dropped. The network or OS silently drops lost datagrams. Inbound datagrams are ignored.
-
Send failures are caught and logged, never fatal.
_on_startcatchesOSError;_on_stopcatches anyException. A failure, such as an unresolvablehost, is logged withlogger.warning(...)plusexc_info=True, and the job proceeds normally. The messages are:Job <name>: failed to send statsd job_started metricJob <name>: failed to send statsd job_stopped metric
-
UDP protocol errors are logged separately. The
statsdlogger records asyncio-level UDP errors surfaced through the datagram protocol asUDP error received: <exc>(see logging configuration for logger names).
The stop metrics are guaranteed to be sent at most once per run, even when two code paths could both reach the stop logic. _on_stop is idempotent: it checks a _stopped flag and returns early if already stopped, setting the flag before any await.
async def _on_stop(self) -> None:
if self._stopped:
return
self._stopped = True
...With concurrencyPolicy: Replace, cancellation and completion may both call _on_stop; only the first call emits metrics. See concurrency and timeouts for the policies and failure detection and retries for how success is computed.
A run canceled during its command gets _on_stop at the end of cancel(). A cancel that arrives while the run's verification check runs terminates the check and does not call _on_stop; wait() calls it once it has collected the check. The stop datagram for that run therefore goes out when the daemon has collected the check, success reflects the check's outcome, and duration runs to that moment.
The
start_timeguard means a forced cancellation also yields a correct duration. The duration is measured from the recordedperf_counterstart to the momentjob_stoppedruns, regardless of how the process ended (normal exit,executionTimeout, orReplacecancellation).
-
Metrics with Prometheus: the pull-side option, a scrapeable
/metricsendpoint on the web API. -
Resource Monitoring: the
monitorResourcessampling behind thecpu/max_rsslines. -
Resource Limits: the memory cap behind the
memory_limit_exceededline. - Configuration Reference: full per-job option list.
- Reporting (Mail, Sentry, Shell, Webhook, Push, Event Log): the other outbound notification channels.
-
Failure Detection and Retries: how
job.failed(and thussuccess:0/success:1) is determined. -
Concurrency and Timeouts:
concurrencyPolicyandexecutionTimeout, which interact with stop metrics.