ADR 0024: Scheduler: recurrence, rain pause and command issuing

Status

Accepted (2026-10-09). Addendum of 2026-10-14 below: the rain rule lives in the Scheduler only; the web app displays its answer.

Context

Phase 3 promises schedules ("weekly, every x days"), a rain pause with a weather service such as Open-Meteo in the backend, schedules for groups, and schedules that rotate between patterns (project plan, feature table and "Mähmuster"). The plan lists a Scheduler service that owns the plans and scales with their number; weather is an adapter port ("Erweiterbarkeit"); schedule.edit exists in the authorization model as can_edit_schedule on account, site, group and mower (ADR 0014). Commands are checked per command kind, recorded and audited by Fleet (ADR 0016); services talk only through the bus and shared contracts, never through each other's databases or internal HTTP.

Five questions had to be answered: how occurrences are computed in a site's time zone, how a schedule fires without bypassing authorization or the command audit, how replicas share the work, where the weather and the location come from, and what happens to a running job when it starts to rain.

Decision

Recurrence is hand-written on zoneinfo. The two kinds we need are "weekly on these days at HH:MM" and "every N days from an anchor date at HH:MM", both in wall-clock time of the schedule's zone. We considered croniter (cron expressions; "every N days" is not expressible, */3 on day-of-month resets at month ends, and DST behaviour depends on the version and on passing an aware start, see the croniter docs) and python-dateutil's rrule (DAILY with interval=N and WEEKLY with byweekday cover both, but it iterates naive wall-clock times and leaves nonexistent and ambiguous times to the caller, see the dateutil docs). Both add a dependency for about forty lines. The Scheduler walks local calendar dates, keeps the dates that match (weekday in the set, or (date - anchor) % N == 0), combines each with the time and converts with fold=0 (PEP 495, zoneinfo). That gives the DST rules we want without special cases: a nonexistent time (02:30 on the spring change in Europe/Berlin) uses the offset before the change and fires at 03:30 the same day; an ambiguous time (02:30 on the autumn change) fires once, at the first occurrence. "Every N days" counts calendar days, so 09:00 stays 09:00 across a change (the interval is 23 or 25 hours that day). The zone is the target's site (Fleet answers it with the target, a cross-site group uses the account's default site) unless the schedule overrides it; without an override the schedule follows a later change of the site's zone at its next firing.

Firing goes through Fleet on behalf of the owner, and authorization is checked twice. Creating or changing a schedule needs can_edit_schedule and can_control on the target (a schedule nobody may run would only skip); the caller becomes the schedule's owner, stored as a PrincipalRef. At every firing the Scheduler checks that the owner still has can_control on each target mower (skipped_permission otherwise) and then sends start_mow through a new core NATS request fleet.commands.issue: IssueCommandRequest{on_behalf_of, account_id, mower_id, command, origin: "scheduler", schedule_id} answered with IssueCommandReply{accepted | error{code, detail}}. Fleet performs exactly the checks of its HTTP endpoint for that subject (relation per command kind, forbidden when the subject sees the mower, not_found when not or when the mower is in another account, service principals refused), records the command with issued_by = the owner plus the new columns origin and schedule_id, publishes it and logs and counts the request (mowdeck_fleet_issued_commands_total{origin, outcome}). So there is one command record and one audit for every command, a revoked right stops schedules at once (both services drop cached decisions on identity.grants.changed), and core NATS stays internal (clients reach services only through Traefik). After MOWDECK_SCHEDULER_MAX_PERMISSION_SKIPS (3) firings in a row where every mower was skipped for missing rights the schedule is disabled and scheduler.schedule.disabled is published; a schedule whose target was removed is disabled the same way. A manual run (run-now) needs can_control and is issued on behalf of the caller. A second request, fleet.targets.resolve, returns the mowers of a mower or group target with the time zone and the coordinates of their sites, so the Scheduler keeps no copy of sites and groups.

Every replica ticks; rows are claimed with FOR UPDATE SKIP LOCKED. A tick (30 s) claims one due schedule (enabled AND next_run_at <= now) per transaction, fires it inside that transaction and stores next_run_at, so two replicas never fire the same schedule and a crash before the commit fires it again at the next tick instead of losing it (a crash after Fleet accepted the start and before the commit can start the mower twice; we accept that rare case over a two-phase protocol, Fleet's command expiry of 30 s keeps it short). The next occurrence is computed from max(now, occurrence): after downtime the Scheduler does not catch up; a firing whose window (start_time plus duration_minutes) has closed is recorded as skipped_window, a late firing inside the window still starts. Each attempt is one row per mower in schedule_runs with an outcome (started, skipped_rain, skipped_permission, skipped_window, skipped_mower_busy, skipped_manual, failed) and published as scheduler.run.recorded. A mower whose latest state (from telemetry.mower.*) is mowing, paused or returning is not started. skip-next marks the next occurrence (skipped_manual). Pattern settings are an opaque dict for StartMow.pattern; an optional list of angles is used round robin per started run (pattern.angle), which implements "rotate between patterns" for native patterns. A group schedule mows all areas of each mower (from its map snapshot); per-mower areas are out of scope.

Weather is a port with Open-Meteo as the default adapter. WeatherProvider.forecast(lat, lon) returns hourly points or None (unknown weather). OpenMeteoProvider asks the forecast API for precipitation (mm, sum of the preceding hour), precipitation_probability (%, chance of more than 0.1 mm in the preceding hour) and rain with past_days=1, forecast_days=2, timeformat=unixtime, no API key, a 5 s timeout, and caches each answer per location rounded to 0.01 degree (about 1 km) for 30 minutes, failures for 60 s. The free API allows 600 calls per minute, 5 000 per hour and 10 000 per day for non-commercial use, and its data is CC BY 4.0 (terms, licence); responses that show Open-Meteo data carry the attribution, and commercial operators need an Open-Meteo plan (the base URL is a setting). StaticWeatherProvider (dry, rain, unknown) serves tests, development and offline installations (MOWDECK_SCHEDULER_WEATHER_PROVIDER=static). A run at t is held when an hour overlapping [t, t + lookahead] reaches the probability or the millimetre threshold, or an hour overlapping [t - cooldown, t] reached the millimetre threshold (defaults 60 %, 0.5 mm, 3 h, 2 h). Without a forecast the schedule's run_when_unknown decides (default: run).

Location: site coordinates, then the map origin, then the last position. Fleet sites get optional latitude and longitude (additive migration, validated ranges, both or none, editable with can_manage_mowers), answered with the target. Without them the Scheduler uses the GeoReference origin of the mower's map, which it keeps from the DEVICE stream (device.mower.<id>.map, a tiny per-mower table like Telemetry's snapshots, together with the map's area ids), and then the last reported position from telemetry. Without any location the rain pause reports no_location and the run goes ahead.

A running scheduled job is sent back on rain or at window end. Started runs stay "open" while their window lasts. Each tick looks at them (SKIP LOCKED as well): when the mower is still mowing and the rain pause now holds (pause_running_jobs_on_rain, default on), or the window closed while it is mowing or paused (return_to_dock_at_window_end, default off), the Scheduler issues return_to_dock through fleet.commands.issue on behalf of the subject that started it. A later state showing the mower docked, or two hours past the window, closes the run.

Consequences

• Fleet stays the only place where commands are authorized, recorded and published; the Scheduler adds two internal request subjects to Fleet and two columns to its command records (both additive).
• Schedules depend on Fleet at firing time; when Fleet does not answer, the run is recorded as failed and the schedule continues with its next occurrence.
• Durable consumers start at new messages, so a Scheduler deployed after a mower's map was published learns the map only with the next snapshot (the Device-Gateway publishes a map when it changed, and again after its own restart); site coordinates and explicit area_ids do not have this gap.
• Open-Meteo needs internet access from the Scheduler; offline installations use the static provider (rain pause effectively off). Another provider is one class implementing the port.
• Not covered yet: per-mower areas for group schedules, maintenance reminders (also a Scheduler task in the plan), planner-generated routes as a schedule target, and the UI (a later task).

Addendum: the web shows what the Scheduler decides (2026-10-14)

A review found the rain rule twice: the mower page had its own default thresholds (60 %, 0.5 mm) and computed the end of a rain pause from the forecast hours, next to domain/rain.py. The rule now exists only in the Scheduler. GET /weather/{mower_id} (and the preview's weather) return, besides wet and wet_until, pause_until, the first time at which the same rule would let a run go ahead (the walk from the decision time past the last rainy hour of the lookahead and the drying time, repeated; null when not holding or when the forecast ends first), and settings, the rain pause settings the decision applied (the schedule's, or the query parameters and defaults). The web app's apps/web/src/lib/mower/rain.ts keeps presentation helpers only (strip bars, "it rains now", the start warning) and reads verdict, hold, pause_until and thresholds from the answer; it holds no defaults of the rule. A change of the rule is a Scheduler change with its tests, followed by uv run poe openapi and pnpm codegen.

---

Mirrored from docs/adr/0024-scheduler-recurrence-rain-pause-and-command-issuing.md at commit 94c46ef36cabac4c3a0bacfc0cc0c0af4defdca3. Edit the file in the repository, not this wiki page.