Scheduled Tasks - developerz-ai/ultimate GitHub Wiki

Scheduled tasks

A task is a cron trigger that enqueues jobs. It never does work itself.

As of 2026-08. Stable API โ€” semver from here (Upgrading).

The canonical shape

// task (cron)
export const nightlyDigest = task({
  cron: '0 3 * * *',
  tz: 'UTC',
  // The occurrence, never the wall clock: a catch-up tick runs long after the instant it fires
  // for, and a payload read off `Date.now()` there is silently for the wrong day.
  enqueue: (occurrenceMs) => [[sendDigest, { occurrenceMs }]],
});

A task only enqueues

If it has a handler body, it is a job, not a task.

Concern Where it lives
When to fire task.cron + task.tz
What to enqueue the enqueue field โ€” (occurrenceMs) => [[jobRef, input], โ€ฆ]
The work the job's run
Retries, steps, concurrency, rate limits the job
Idempotency the job's idempotencyKey โ€” this is what absorbs a double fire
Observability of the run the job's spans and x jobs show <id>

enqueue must be cheap and synchronous-shaped: build inputs, return the list. Fanning out per tenant is a job that enqueues children, not a loop with a DB read inside enqueue.

export const rotateApiKeys = task({
  cron: '15 2 * * 1',
  tz: 'Europe/Berlin',
  enqueue: () => [[fanOutKeyRotation, {}]],
});

The fluent surface

Every projection is a method on the task โ€” nightlyDigest.enqueue(), never runTask(nightlyDigest) โ€” and every declared field is lifted onto it. A task has no .def.

Member Is Rule
nightlyDigest.entries(occurrenceMs?) the [[jobRef, input], โ€ฆ] pairs this task fires โ€” the handle's read of the declared enqueue field built for that occurrence, not for "now" โ€” a catch-up dispatch runs long after the instant it fires for, and a payload derived from the wall clock there is silently for the wrong day. Defaults to now, the honest answer for the two callers that have no occurrence: a manual enqueue() and describe()
.enqueue(options?) fire the declared entries immediately the backfill and "run it again" path โ€” no scheduler, no leader, no tick. Goes through the same facade <job>.enqueue uses, so it joins the caller's transaction on the same terms. Returns one { job, result } per entry
.describe() the manifest row kind, name, cron, tz, catchUp, maxCatchUp, jobs โ€” the job names, in declaration order, because a task's entries are a sequence and not a set
.kind .name .cron .tz .catchUp .maxCatchUp the declaration, lifted readable, and already resolved: kind is 'task', catchUp defaults to 'skip', maxCatchUp to 10

A task reaches its jobs only through its own entries, and those entries carry job handles, never job names. So the scheduler dispatches with <job>.idempotencyKeyFor(input) and <job>.retry.attempts read off the handle โ€” a task never restates a job's retry policy and never resolves a job by string.

Manual enqueue() and a scheduled tick differ in exactly one place: the key. A tick's is occurrence-scoped, <task>:<occurrenceMs>:<the job's own key>, so two schedulers or a retried tick cannot double-fire that occurrence. enqueue() uses the job's plain key, because a manual run has no occurrence to scope to โ€” and scoping it to whichever occurrence it happened to land in is the one thing that would make a backfill dedupe against a real tick.

Fields

Field Required Rule
cron yes standard 5-field expression. No seconds field, no @hourly aliases โ€” one way to write it
tz yes explicit IANA zone ('UTC', 'Europe/Berlin', 'America/New_York'). Omitting it is a compile error
enqueue yes (occurrenceMs) => [[jobRef, input], โ€ฆ]. Zero or more pairs; an empty list is a valid no-op tick. Read back through the handle as entries(), never as .enqueue โ€” that name on the handle is the fire method
catchUp no โ€” default 'skip' what to do when the scheduler was down across one or more occurrences. 'skip' collapses them into one dispatch for the latest missed occurrence and drops the older ones, 'run-once' fires a single catch-up for the earliest missed one, 'run-all' fires one per missed occurrence
maxCatchUp no โ€” default 10 how many occurrences one round of 'run-all' fires; the rest wait for the next round. 'skip' and 'run-once' fire ONE occurrence whatever the gap โ€” the latest missed and the earliest โ€” so neither is capped by it
name no the export name, stamped by defineApi({ tasks: [scheduledTasks] }). A module nobody hands over keeps anonymous-task-<n>; a definition carrying its own name: keeps that

Nothing else. There is no timeout, no retry, no concurrency on a task โ€” those belong to the job it enqueues. No queue either: the queue is the job's, and a per-call override rides on the fire, as <task>.enqueue({ queue }).

tz is required for a reason

Without explicit tz Consequence
Server-local time the schedule silently moves when the container's TZ changes or a region differs
Fixed offset (+02:00) drifts by an hour twice a year against "3am local"
DST spring-forward gap a 2:30 local tick has no instant โ€” the framework fires it at the zone's next valid instant
DST fall-back overlap a 1:30 local tick occurs twice โ€” a fixed time (30 1 * * *) fires once, on the first occurrence; an interval (*/5 * * * *, 0 * * * * โ€” minute or hour field * or */n) runs through both passes, because the repeated hour is real elapsed time

tz is checked against the runtime's own IANA database, not merely for non-emptiness As of 2026-08. The declaration asks Intl.DateTimeFormat to resolve the zone, and Intl carries the runtime's copy of the tz database โ€” the only check that can tell America/Bogota from Bogota. So a non-empty string is not a timezone:

X_INVARIANT: task "nightlyDigest" has tz "Bogota", which is not a zone in the IANA tz database
fix: use the full zone id on task("nightlyDigest"), e.g. tz: 'America/Bogota' โ€” list the valid
     ones with: bun -e "console.log(Intl.supportedValuesOf('timeZone').join('\n'))"

It fails at declaration, not at the first tick. The alternative is what an abbreviation used to do: resolve every occurrence in UTC and run the cron five hours off, silently, forever.

Store UTC, schedule against an explicit zone, format at the edge. See Timezones and dates.

The scheduler role

Fixed 1. Cron dispatch only.

Property Behavior
Leader election an expiring lease row in x_scheduler_leader (postgresLeaseLeader), never pg_try_advisory_lock โ€” that grant belongs to the session, not to the process: it outlives every transaction, survives or dies by the pool's reset policy on release, and either way no node can renew it or prove it still holds one. A stranded lock nobody can release and a lock silently dropped mid-round are both wrong for election. Whoever holds an unexpired row dispatches
Renewal acquire() is the renewal, asked every round โ€” tick 1s, lease TTL DEFAULT_LEADER_TTL_MS = 30s. A node that stops renewing loses the lease by expiry, with nothing to clean up
Second instance a warm standby, not a duplicate. It calls acquire() every round like the leader does; while another node holds an unexpired row it acquires nothing and dispatches nothing, and the round after that row expires it takes the row and becomes the leader
Leadership lost logged once as jobs.scheduler.leadership-lost; the node keeps polling and takes over on the first round after the holder's lease expires. The process does not exit, and neither does a replicator container that loses its lock: it stands by unready and re-asks. Only x dev --role replicator refuses (X_REPLICATOR_SLOT_HELD)
/readyz on the standby there is none. ROLE=scheduler binds the metrics port only (/metrics, METRICS_PORT, default 9090) and no HTTP server, so liveness is that endpoint โ€” the probe docs/ops/01-kubernetes.md declares. Nothing routes request traffic to a scheduler in the first place
Missed tick (leader down, node paused, clock jump) fires late rather than being skipped
Double fire during handover absorbed by the enqueued job's idempotencyKey โ€” the second enqueue returns the existing handle, no new row
Drain on SIGTERM registered at the accept phase: stop dispatching, wait out the dispatch round in flight, then delete the lease row โ€” the standby promotes on its next round rather than waiting out the TTL, and never onto an occurrence this node is still enqueueing for
Two dispatch rounds at once impossible in one process. The timer re-arms on the round it finished, and any other caller joins that round instead of opening a second one over the same last-tick state
Durable state none in the process. Last-tick state is a Postgres row

That chain is the whole safety argument: at-least-once dispatch + required job idempotency = effectively-once work. A scheduler that guarantees exactly-once dispatch does not exist; one that guarantees never-silently-skipped does.

Role table and drain sequence: Deployment.

Introspection

Command / tool Output
x tasks list --json the descriptor โ€” name, cron, tz, catchUp, maxCatchUp, jobs โ€” plus the resolved next occurrence, rendered in the task's own tz, and its nextMs epoch instant; and the last fire (last, lastMs, lastFiredAtMs โ€” null before the first), read from the queue, which x tasks opens when a task is declared. All derived, never declared fields
x tasks show <name> --json the same, plus the cron in words (describe) and the next N occurrences (--count, default 5)
/admin/jobs/tasks the jobs dashboard every defineAdmin() serves: schedule, zone, last fire, next fire (nextTaskRun), paused; pause, resume and run now behind job:manage โ†’ Admin dashboard
MCP admin.x_job_tasks.list, admin.action.job.task.run-now the same screen and actions as admin MCP tools, same authz (job:read, job:manage)
/_x dev panel the jobs tab: the dev process's tasks beside its queues and runs
x.manifest.json generated tasks section โ€” the build-time source of truth
<task>.describe() in process: kind, name, cron, tz, catchUp, maxCatchUp, and the job names it enqueues
<task>.entries(occurrenceMs?) in process: the exact pairs that occurrence fires, inputs included โ€” which describe() drops
$ x tasks list --json
{"ok":true,"command":"tasks","summary":"1 task(s)","findings":[],"data":[
  {"kind":"task","name":"nightlyDigest","cron":"0 3 * * *","tz":"UTC","catchUp":"skip",
   "maxCatchUp":10,"jobs":["sendDigest"],
   "nextMs":1786503600000,"next":"2026-08-12T03:00:00Z"}]}

next carries the task's own zone offset, never a machine-local one: the same 0 3 * * * in America/New_York reads 2026-03-06T03:00:00-05:00 before the spring-forward and 2026-03-09T03:00:00-04:00 after it.

Testing

Frozen clock. clock.advance drives cron โ€” never sleep, never wall-clock.

test('nightlyDigest enqueues one digest job per day', async ({ seed, clock }) => {
  await seed('one-org');
  clock.advance('3d');
  const trace = await runJobs.drain();
  expect(trace.enqueued(sendDigest)).toHaveLength(3);
});
Asserted Why
Tick count over an advanced window the cron expression means what you think in the declared tz
Idempotency absorbs a replayed tick advance to the same instant twice; the job row count must not change
DST boundary advance across a spring-forward and a fall-back in the task's zone
enqueue returns the right inputs the pairs, not the work

Runner: x test job (tasks are dispatch, jobs are the assertion surface). See Testing.

Errors

Code Cause Fix
X_IDEMPOTENCY_REQUIRED a task enqueues a job with no idempotency key โ€” a double fire would duplicate work add idempotencyKey to the job
X_FORBIDDEN the task's system actor lacks a permission the enqueued action requires grant it explicitly; a task never bypasses policies
X_DRAINING a tick landed while the leader was releasing its lock none โ€” the standby promotes and the tick fires late

Full index: Error codes.

Rules

  • A task never contains a handler body. If it does work, it is a job.
  • tz is always explicit and always IANA.
  • Every job a task enqueues must be idempotent โ€” that is what makes leader handover safe.
  • Never rely on a task firing at an exact instant. Rely on it firing.
  • Never use a task as a poller for something an action could publish. Cron is a fallback, not an event bus.
  • One task per schedule. Two schedules for one job is two tasks, not a conditional inside enqueue.
โš ๏ธ **GitHub.com Fallback** โš ๏ธ