Background Job Health Design - edmozley/freeitsm GitHub Wiki

Background job health - design (#182)

Status: built (19 Oct 2026), in the next release. This page sets out what I am building for issue #182, and why. It was written before any code so the approach could be checked first. It is designed alongside #181, and section 4 says how the two fit. The user guide is Background jobs.

Updated 19 Oct 2026: @Dr4inder agreed all six recommendations in section 7, and asked for two health rules and one wording to be made explicit - see section 9. In short: only a scheduled run keeps a job fresh, so page loads and Run now can never keep the health address green; and a skipped, refused or dry run can never clear a failure.

Since then (4.8.0, #233): FreeITSM now has a built-in scheduler - one scheduled task, cron/run.php, starts every job, and the Docker image runs it itself - so the Docker remarks below describe the time before it. The health rules here are unchanged: scheduler runs count as cron, the optional fallback's runs count as page and never keep a job healthy. A job's schedule on the tab is now also what health expects of it. See Scheduled tasks and the developer guide.

The request, from issue #182 (raised by @Dr4inder):

"A scheduler outage can otherwise leave the application looking quiet while expected work, notifications or synchronisation never occur."

In plain words

Several parts of FreeITSM rely on background jobs that the server runs on a timer - a cron entry on Linux, a scheduled task on Windows: sending SLA warnings, delivering webhooks, creating repeating tasks, syncing calendars and directories, checking domains. If the timer stops - a server rebuild, a moved install, a Docker container with no scheduler - nothing tells you. No error appears; things just quietly stop happening. The first sign is usually somebody asking why an SLA warning never came.

The request is for one place that shows every background job: which ones this install needs, when each last worked, whether any is overdue, failing or stuck - plus a health address an outside monitoring tool can check, so an alarm can go off without relying on FreeITSM itself.

1. The principle: measure, never schedule

FreeITSM does not become a scheduler. The jobs keep running exactly as they do now - from cron, Windows Task Scheduler, or a monitoring tool calling their web address. This work only records each run and reads those records.
"Last success" means the work finished. A job that starts and crashes, is turned away by its lock or minimum interval, fails its token check, or is a dry run, never refreshes it. A run that finds nothing to do is a success.
Only the schedule proves the schedule. A job that is meant to run unattended is fresh only when its scheduled path - the command line or its token web address - has succeeded recently. A page load or Run now does the work, but it never makes an overdue schedule look healthy (section 9).
Nothing but success clears a failure. A skip, a refused call or a dry run is not evidence that the problem has gone, so it can never turn a failing or partial job healthy again (section 9).
Health is worked out when you look. Whether a job is overdue is calculated at the moment the page or the health address is opened - so a stopped job shows as overdue because it has stopped, not because it ran again to say so.
Honest about what it cannot see. FreeITSM cannot inspect the server's crontab. A job with no recorded run is "never seen" - not "the cron entry is missing", which it cannot know.

2. What FreeITSM records today

Mapped from the code (3.9.0). There are 13 background jobs and no shared way of recording them:

Job	What depends on it	What it records about its own runs
SLA breach check	SLA warning and breach emails	✅ A proper run log (`sla_cron_runs`): start, finish, outcome, counts
Webhook deliveries	Outbound webhooks	⚠️ A "last run" time written before the work
Scheduled workflows	Contract and warranty expiry workflows, planned maintenance, project alerts and digests	⚠️ Written before the work; failures in its parts are swallowed and it still says OK
Integration poll	Jira / tracker status and comments	⚠️ Written before the work; errors land on each linked ticket, not the connection
Repeating tasks	Tasks on a fixed schedule	⚠️ Written before the work
Attachment reading	Searching inside attachments	⚠️ Written before the work
Calendar sync pull	Changes made in analysts' calendars	❌ Nothing at job level
Domains	Lookups, checks, alerts	⚠️ One time, written by the cron and by page loads and Run now alike
LMS reminders	Training reminder emails	⚠️ Same - cron, page loads and Run now look identical
Proxmox sync	Proxmox inventory	⚠️ Per server only
VMware Cloud Director sync	vCloud inventory	⚠️ Per server only
Documents maintenance	Document search, file tidy-up	❌ Exit code only
Directory sync	LDAP / AD / CardDAV people	✅ A run log per provider (`directory_sync_runs`)

The useful pieces that already exist - the SLA run log, the directory-sync per-provider log, the webhook page's running / stale / down judgement, the per-server sync status columns - are the model for what follows. Reporting → System logs describes itself as covering "background processes, scheduled tasks", but no job writes to it yet.

Docker: the image runs no scheduler at all, so on a fresh Docker install every job is "never seen" until the host is set up to run them - worth saying plainly on the page and in the docs.

Not a background job: fetching email into tickets is done by the browser while the ticket list is open, not by a scheduled task. It has its own mailbox health warnings already, and it stays outside this work (section 8).

3. The proposal, in six parts

Part A - One run recorder for every job

A new job_runs table and a small shared helper that every job calls at the start and the end of its work. One row per run:

Column	Holds
job	which job (a fixed key such as `task_recurrence`)
trigger	cron (command line), web (its token address), page (run in passing by a page load) or manual (Run now)
started / finished	UTC times, and the duration
outcome	ok, partial (some sources failed), failed, skipped (lock, minimum interval, switched off, or a dry run), or running while it works
counts	what it did - "12 sent, 0 failed", "3 occurrences created"
summary	a short message written by the job, never a raw error that could carry a password or token

A call turned away at the door - a wrong or missing token - is not written as a run: anybody can make one, and a table that strangers can fill is not a record. The job keeps one line instead, "last refused call: wrong token, 14:02", shown on the page, so a scheduler still calling the old address after a token change is easy to spot. It never counts as a run of any kind.

Jobs that work through several sources - tracker connections, directory providers, analysts' calendars, Proxmox and vCloud servers, domains - also write one source result each (job_run_sources: which source, ok or failed, a short message). That is the issue's point that "partial failures should remain visible rather than being hidden by an overall successful invocation": one failing directory provider makes the run partial, named, even though the other three worked.

The SLA job's existing log becomes the first user of the shared recorder; its own table stays for its own screen. The existing "last run" settings stay too, so nothing that reads them breaks - but health no longer reads them, because five of them are written before the work starts.

Retention: ok, partial and failed runs are kept for 13 months by default, long enough for a yearly audit and for #181's report (section 4); skipped runs, which can arrive every minute, for 30 days.

Part B - What each job is expected to do

A job registry - one file, like the currency list - naming every job with:

what depends on it, in plain words;
when it is needed: the same test the job itself uses - repeating tasks only when a series runs on a fixed schedule, the tracker poll only when a connection is active, LMS reminders only when they are switched on;
its expected interval (from the docs: SLA 5 minutes, webhooks 1 minute, repeating tasks daily...) and a grace period;
its longest normal run, beyond which a run still marked running is stuck.

An administrator can change a job's expected interval, because installs schedule differently, and can mark a job not used here on purpose.

Each job's health is then worked out on reading:

Health	Means
Not needed	The feature it serves is not in use on this install
Not used here	An administrator has said so
Never seen	Needed, but no scheduled run has ever succeeded - the page says whether page loads or Run now have run it
Healthy	Last scheduled success within its interval plus grace, and nothing failing
Overdue	Last scheduled success older than that - even if a page load or Run now has run it since
Failing	Its most recent real run failed (the last success is still shown)
Partial	A source failed and has not succeeded since - named
Stuck	A run started longer ago than its longest normal run and never finished

A job can have more than one problem at once - failing and overdue - and the page lists them all. The headline is the most serious.

The rules behind the table, which section 9 explains:

Freshness reads only scheduled runs. Each job keeps two times: the last scheduled success (command line or token web address) and the last success of any kind. Health and ?strict=1 use the first. The page shows both, so "last scheduled run 9 days ago; last page-load run 2 minutes ago" is plain to see.
Failure reads only real runs. "Failing" and "partial" are worked out from runs that actually processed work - outcome ok, partial or failed. Skipped runs (lock, interval, switched off, dry run) are kept in the history but never consulted, so the newest run being a skip changes nothing.
Recovery is per source. A source is failing until a later run processed that source successfully. A job whose tracker connection "Jira" failed stays partial until Jira is polled successfully - not merely until some other run succeeds - or until Jira is switched off or removed, which changes whether it is expected.
Saying "not used here" is the explicit change. Marking a job not used here, or the feature behind it no longer being in use, clears its health because it is no longer expected - and the page records who did it.

Page loads are not proof of a schedule. Domains, LMS reminders and project alerts also run in passing when somebody opens their pages, which used to keep a job looking alive while its cron entry had gone. Under rule 1 they still do the work, and are shown in the history as page runs, but they can never keep the job healthy.

Part C - The page

Reporting → System logs → Background jobs - a third tab beside User logins and Email imports, where the logs already live.

At the top, the health of every job: name, what depends on it, health, last success, last run and its outcome, expected interval, and how it was triggered. Unhealthy jobs first.
Below, the run history: every run, filterable by job, outcome and period, each opening to its counts, message and source results.
For "never seen" and "overdue", a short note on what to check - with a link to Scheduled tasks, which says how to set each one up on Linux, Windows and Docker.

Who can open the tab is a decision for section 7.

Part D - Being told

A log page only helps somebody who looks at it, and the point of the issue is that a stopped job is silent. So:

a Watchtower card, shown only while a job is unhealthy, naming the jobs and what they affect;
a notice on the System page for administrators, in the same way as the other System warnings.

Both are worked out when the page is opened, so neither depends on the scheduler being alive.

Part E - A health address for outside monitoring

GET /api/v1/jobs/health on the REST API, using an API key that holds a new "Background job health - read" permission. It returns, for every job: its health, last success, last run and outcome, expected interval, and the summary and failing sources - and an overall status.

It never runs a job and never returns passwords, tokens or connection details.
For simple uptime monitors, ?strict=1 answers HTTP 503 whenever anything needed is unhealthy, and 200 otherwise - so a tool that only looks at status codes still raises the alarm.
If FreeITSM or its database is down, the address fails - an error, never a cheerful "healthy" - which is itself the alarm (acceptance criterion 6).
Job health is install-wide, so a key that can see only some companies is not enough: the key's analyst must be an administrator or reach every company.

Part F - Fixes found while mapping it

These would make the health page wrong, so they come first:

Five jobs write "last run" before doing any work (webhooks, scheduled workflows, the tracker poll, repeating tasks, attachment reading) - so a crashed or empty run looks healthy. The recorder replaces this for health.
The scheduled workflows job swallows failures in its parts (planned maintenance, project alerts, digests) and still says OK. Each part becomes a source result.
The SLA job records a run that sent some emails but hit errors as plain success. It becomes partial.
LMS reminders can never be run by web address: its token is never created, and is not read back correctly. Fixed, and added to the token tool.
Tracker poll errors are written to every linked ticket rather than to the connection, and System → Integrations does not show when a connection last polled.
Calendar sync's health treats "every pull is failing" the same as "never run", and warns on installs that only use subscription links.
Two warnings say mail is collected by a scheduled task, which does not exist.
Calendar sync, Proxmox and vCloud jobs have no protection against overlapping runs. (Proxmox and vCloud since fixed: their crons take a database lock, GET_LOCK('freeitsm_proxmox_sync') / 'freeitsm_vcloud_sync'.)
The docs disagree on how many jobs there are ("nine", "four") and on intervals; Scheduled tasks is missing four jobs and has no Docker section.
Jobs that fail exit with success (webhooks, workflows, tracker poll), so a scheduler cannot tell either. They will exit non-zero.

4. How this fits with #181

The issue makes the point itself: "a healthy scheduler does not prove that a control was completed, and a completed task does not prove that the scheduler is healthy." The two designs are built to answer each other:

#181's report can say what the records show. It shows a repeating task's occurrence as Missing or Created late. With #182's run history for the repeating-tasks job, it can add what the run history says about that period: "No recorded run of the repeating-tasks job between 2 and 9 March", "The job failed on 3 March", or "Created on 9 March by a catch-up run". That is the auditor's next question, answered on the same page. It says "no recorded run", never "did not run", and "run history not available" before recording began or beyond retention - an absence of records does not prove the scheduler stopped (section 9).
#181's history links to the run. When the scheduler creates an occurrence, #181's task history records created by the repeat - on schedule / catch-up; it now also records which job run created it.
Retention is set for audits. Run history is kept for 13 months so a yearly evidence report can still read it.

#181's design has been updated with these three points. Build order: #182's recorder (Part A) should land before #181's report, so the report has run history to read.

5. The issue's acceptance criteria

#	Criterion	Where
1	Identify the required jobs, their cadence and last successful processing	Parts B, C
2	A stopped repeating-task scheduler turns unhealthy after its interval and grace, with no new record	Part B (worked out on reading)
3	Never seen, overdue, failed, partial, stuck and intentionally inactive are distinguishable	Part B
4	Skipped and failed runs keep the previous success; empty successful runs count	Parts A, B
5	Diagnostics name the job or source without secrets	Part A (summaries written by the job, source results)
6	An outside monitor can detect unhealthy jobs on its own	Part E
7	Existing cron, command-line and web paths keep working; no second scheduler, no duplicate runs	Section 1, Part A

6. The order to build it in

Record - the recorder and registry (Parts A, B) and the fixes (Part F). Every job starts writing proper history from this release.
See - the Background jobs tab (Part C), the Watchtower card and System notice (Part D).
Monitor - the health address and its permission (Part E).

Phase 1 also unblocks #181's report (section 4).

7. Decisions

All six recommendations were agreed on the issue on 19 Oct 2026.

#	Decision	Agreed
1	Where the page lives	Reporting → System logs → Background jobs, beside the existing log tabs - with the Watchtower card and System notice so nobody has to go looking.
2	Who can open it	A new permission, "View background job health", rather than everyone with Reporting: job messages name mailboxes, directories and servers. It also fits #181's Auditor role.
3	Expected intervals	Built-in defaults from the docs, changeable per job by an administrator, plus not used here.
4	Telling somebody by email or the bell when a job turns unhealthy	Not in the first version. Sending that message would itself need a background job - the very thing that may have stopped. The health address and an outside monitor are the reliable alarm; Watchtower and the System notice cover people already in FreeITSM.
5	Retention	13 months for real runs, 30 days for skipped ones - long enough for a yearly audit.
6	Docker	Document the host's cron for Docker installs in Scheduled tasks. A scheduler inside the container could come later, as its own change.

8. Deliberately out of scope

No second scheduler and no new way to run jobs - the existing Run now buttons stay as they are.
No monitoring service, database or job framework - the issue asks for none, and outside tools consume the health address.
Email fetching stays browser-driven with its existing mailbox health warnings; it is not a background job today.
Nothing here proves that the work a job should have produced was done - that is #181's job, for repeating tasks.

9. Three clarifications (19 Oct 2026)

#	Asked for	What changed	Where
1	Scheduled freshness. For a job expected to run unattended, track the last successful scheduled invocation separately from page and manual runs, so "healthy, but only page loads have run it" cannot leave a status-code-only monitor green while the schedule is overdue. Once the scheduled path passes its interval plus grace, overall health and `?strict=1` must say so - even if somebody opened a page or clicked Run now a moment ago - for both the command line and the scheduled web address.	Freshness reads only runs triggered by cron (command line) or web (the token address). Page and manual runs are recorded and shown, but never refresh it. An overdue schedule is Overdue whatever ran since, and `?strict=1` answers 503. The "healthy, but only page loads" state is gone.	Section 1, Part B rule 1
2	Skipped attempts after failures. A later lock or interval skip, a refused call or a dry run must not clear a failing or partial state just by being the newest run. Recovery needs successful processing of the affected work or source, or an explicit change to whether it is expected.	Failure is worked out only from runs that processed work, so skips are never consulted. Refused calls are not runs at all. Partial is tracked per source: a source stays failing until a run processes that source successfully, or it is switched off or removed. Not used here is the explicit change, and is recorded.	Part A, Part B rules 2-4
3	Wording for #181. Say "no recorded run in this period" unless the records establish the cause more specifically - an absence of records, particularly before recording began or outside retention, does not prove that cron did not run.	#181's report uses that wording, and says "run history not available" for periods before recording began or beyond retention. Recording's start date is kept, so the report can tell the two apart.	Section 4, #181 Part E

10. A failed lookup is not an answer (20 Oct 2026)

@Dr4inder reproduced that when every database read failed, the health check said healthy - every job Not needed - because each failed lookup quietly became no. Now a lookup that fails (anything but a table that does not exist yet) makes the job Unknown, which is unhealthy, and the strict address answers 503. Not needed and source removed need a lookup that succeeded and said so. Tests cover a failed requirement lookup and a failed source-health read, among others. Details: developer guide.

Scheduled tasks - how each job is set up
Recurring task evidence - design (#181)
Webhooks · REST API