Background Jobs - edmozley/freeitsm GitHub Wiki
Reporting → System logs → Background jobs shows every scheduled job FreeITSM relies on, whether each one is doing its work, and every run it has made. It answers one question that nothing else in FreeITSM could: has anything quietly stopped? From 4.8.0 it is also where you set when each job runs, for the built-in scheduler (§6).
Why this exists. A background job that stops does not announce itself. Nothing errors and nothing appears in a log - SLA warnings just stop arriving, webhooks sit in a queue, repeating tasks stop appearing. The first sign is usually somebody asking why. This page, a card on Watchtower and a notice on the System page make that visible, and a health address lets an outside monitoring tool raise the alarm on its own. Requested in #182; the design is Background job health - design.
- Setting the jobs up is a different page: Scheduled tasks lists every job, what it needs and how to schedule it on Windows, Linux and Docker.
- Developers: Background jobs - Developer Guide.
At the top, a box says whether the built-in scheduler is running:
| The box says | Means |
|---|---|
| The built-in scheduler is running (green) | One scheduled task, or the Docker container, starts every job below |
| ...last ran at ... and has stopped (amber) | It was running and has not for a few minutes: check the scheduled task still runs every minute. The line to schedule is shown |
| Each job below needs its own scheduled task | No built-in scheduler yet: the line to schedule is shown, with the fallback switch below it (Scheduled tasks) |
| This container runs the built-in scheduler itself, but it has not run | Docker only: check docker logs, and that FREEITSM_SCHEDULER is not set to 0 |
| No timezone is set for this container | Docker only: daily times are in UTC until you set TZ
|
The table below it lists the jobs, the ones that need attention first. Each shows what stops without it, its health, the last time its schedule ran it successfully, its last run of any kind, and how often it is expected to run.
| Health | Means | What to do |
|---|---|---|
| Healthy | Its schedule ran it successfully within its expected interval plus some slack | Nothing |
| Overdue | No scheduled run has succeeded for longer than that | Check the job is still scheduled - see Scheduled tasks |
| Failing | Its most recent real run failed - the message says why | Read the message; the run history has the detail |
| Partial | Part of its work failed and has not succeeded since - one tracker connection, one directory, one server, one calendar. It is named | Fix that source; the job recovers when that source next works |
| Stuck | A run started longer ago than any normal run takes, and never finished | Look for a hung process; the next finished run records the stuck one as failed |
| Unknown | FreeITSM could not read something it needs to judge the job - whether it is needed, its settings, its run history or how its sources last ran. Usually the database being unavailable. Shown as a problem, and the strict health address answers 503, rather than assumed fine | Check the server's error log |
| Never seen | The job is needed here, but no scheduled run has ever been recorded | Set up the built-in scheduler (the box at the top shows the line), or schedule the job on its own |
| Not needed | The feature it serves is not in use on this install - no fixed-schedule repeats, no tracker connections, LMS reminders switched off | Nothing |
| Not used here | Somebody has said this install does not run it, on purpose | Nothing - who said so, and when, is shown |
Worked out when you open the tab. Nothing has to run for a job to turn overdue - time passing is enough. That is the point: a job that has stopped cannot tell you it has stopped.
After upgrading, every job you need shows Never seen until its schedule next runs it, because run history starts on the day this was added. A job you run every few minutes turns healthy within minutes; a daily one, by tomorrow.
Pushed sources (#199): each Hyper-V connection is listed as Hyper-V reports: . It has no command for FreeITSM to run - it is healthy while the collector's reports arrive within the connection's own interval and grace, and partial when the last report could not read everything. See Hyper-V discovery.
Three rules decide health. They were agreed on the issue, and they are what make the page trustworthy rather than reassuring.
- Only the schedule proves the schedule. Some jobs also run when somebody opens a page (Domains, LMS reminders, project alerts) or presses Run now. Those runs do the work, and they appear in the history - but they never keep a job healthy. A job whose cron entry has gone turns overdue however often somebody opens its page, and the tab says "It has run since, but only by page load".
- Nothing but success clears a failure. A run that did no work - turned away by its lock or minimum interval, switched off, or a dry run - is shown as Skipped, and never makes a failing job look healthy, however recent it is.
- A failing source stays failing until that source works. If one of four directories fails, the job is Partial and names that directory until a run processes that directory successfully - not merely until some other run succeeds. Switching the source off, or deleting it, also clears it: it is no longer expected.
A call turned away at the door - the job's web address called with the wrong token - is not a run at all, because anyone can make one. The job shows "Last call turned away: wrong token" instead, which is the clue when a scheduler is still calling an old address after the token changed.
The bottom half lists every run, newest first, filtered by job, outcome, what started it and period. Click a run to see its counts, its message and the result for each source it worked through.
| Started by | Means |
|---|---|
| Schedule (command line) | the built-in scheduler, or cron or Windows Task Scheduler running the script |
| Schedule (web address) | a scheduler or monitoring tool calling the job's token address |
| Page load | the job ran in passing when somebody opened its page, or the fallback started it from somebody's open page |
| Run now | somebody pressed Run now (or Sync now, or ran one directory by hand) |
| Outcome | Means |
|---|---|
| OK | It worked. A run that found nothing to do is OK |
| Partial | Some sources worked and some did not |
| Failed | It did not do its work |
| Skipped | It did no work: another run of it was still going, its minimum interval turned it away, the feature is off, or it was a dry run |
| Running | It is running now |
Kept for 13 months, so a yearly audit can still read it - repeating task evidence uses it to explain missing occurrences. Skipped runs, which can arrive every minute, are kept for 30 days.
Messages never contain secrets. A job's summary is written by the job, and anything that looks like a password, a token or a URL's query string is blanked before it is stored.
A page only helps somebody who looks at it. So while anything needs attention:
- Watchtower shows a Background jobs card above the others, naming each job and what it affects. It appears only while something is unhealthy, and only for people allowed to see job health.
- The System page shows an amber notice to administrators.
Both are worked out when the page is opened, so neither depends on any job running.
Why no email or bell notification? Sending one would itself need a background job - the very thing that may have stopped. The reliable alarm is an outside monitor (below); Watchtower and the System notice cover people who are already in FreeITSM.
GET /api/v1/jobs/health
GET /api/v1/jobs/health?strict=1
Call it with an API key holding the Background job health - read permission (System → API). It returns every job's health, last scheduled success, last run and failing sources, and an overall healthy or unhealthy. It never runs a job and never returns passwords, tokens or command lines.
-
?strict=1answers HTTP 503 while any needed job is unhealthy, and 200 otherwise - so a monitoring tool that only looks at the status code still raises the alarm. - If FreeITSM or its database is down, the address fails - an error, never a cheerful "healthy" - which is itself the alarm.
- Install-wide, so the key must be too. Job health covers every company, so the key, and the analyst it acts as, must reach every company (or the analyst must be an administrator).
Examples - a check every five minutes from Uptime Kuma, Zabbix, Nagios or a plain script:
curl -fsS -H "Authorization: Bearer fitsm_..." "https://itsm.example.com/api/v1/jobs/health?strict=1" > /dev/null \
|| echo "FreeITSM background jobs need attention"The full reference is on REST API.
Edit on a job (with the permission below) sets its schedule, which is also what its health expects - one setting, so the two can never disagree:
- Every so many minutes - leave it empty for the default from Scheduled tasks: the SLA check every 5 minutes, repeating tasks daily, and so on. The slack before it turns overdue grows with the interval.
- Once a day, at a set time - in the install's timezone, shown beside the time. A job has nobody signed in, so nobody's own timezone applies. Its health expects it once a day.
- Off - not used here - for a job this install deliberately does not run. The scheduler never starts it, its health is no longer checked, and the tab records who marked it and when, because it silences that job's alarm.
With the built-in scheduler, the Expected column also shows each job's schedule and its next start. Without it - each job on its own scheduled task - set this to match how often that task runs, or the job will show overdue between runs.
The fallback switch sits under the scheduler box when no scheduler is running: Start due jobs while somebody has FreeITSM open. Off by default. See Scheduled tasks for what it does and does not do. Hyper-V reports keep the older form - an expected interval and not used here - because FreeITSM does not run them.
| Permission (System → Roles) | Lets someone | Notes |
|---|---|---|
| View background job health | Open the tab, see the Watchtower card | Also needs Reporting module access and access to every company - job messages name directories, servers and tracker connections that belong to all of them |
| Change what background jobs are expected to do | Set each job's schedule, mark jobs not used here, switch the fallback on or off | Sensitive: it can silence an alarm |
Administrators hold both. Nobody else sees the tab until a role grants it, so upgrading changes nothing for them.
From 4.8.0 the image runs the built-in scheduler itself, so the box at the top should be green within a minute of the container starting. Set TZ for daily times in your own timezone. Before 4.8.0 the image ran no scheduler, and each job had to be scheduled from the host - see the Docker section of Scheduled tasks.
- Scheduled tasks - setting each job up
- Background jobs - Developer Guide
- Background job health - design (#182)
- Repeating task evidence - which reads this history
- Reporting · Watchtower · REST API · Roles and permissions