Monitors - ThinkHumanDotDev/marmot GitHub Wiki

Monitors

A monitor is one thing Marmot checks on a schedule: a URL, a host and port, a DNS name, a database, or a system that reports in by itself. Each check produces a heartbeat (up, degraded, down, pending or maintenance, with a message and a response time), the heartbeats feed the uptime statistics and the status pages, and status changes trigger notifications. Monitors belong to an organization; viewers can see them, members and above can create, edit, pause and delete them (Organizations and members).

Types

The type decides what a check does and which fields the form shows. The built-in set on main:

Group Type UP when
General HTTP(s) the response status is in acceptedStatusCodes (default 200-299)
General HTTP(s) - Keyword the response body contains the keyword (or lacks it with invertKeyword)
General HTTP(s) - Json Query a JSONata expression over the JSON response compares as expected
General TCP Port a TCP connection to hostname:port succeeds
General Ping the host answers ICMP echo requests
General DNS the resolver returns a record of dnsResolveType for hostname
Passive Push your system called /api/push/<token> on schedule (interval or cron, plus grace)
Passive Manual you set the status by hand; nothing is checked
Special Group its children are UP by its aggregation mode (worst, quorum or weighted)
Special Docker Container the container is running (and healthy, when it has a health check)

HTTP monitors also support request method and body, extra headers, redirects, ignoreTls, basic/bearer/ OAuth2 client-credentials/NTLM/mTLS authentication and certificate-expiry alerts. The extended set adds gRPC, WebSocket, MQTT, Kafka, RabbitMQ, SMTP, SNMP, NTP, SFTP, RADIUS, Tailscale ping, MySQL/MariaDB, PostgreSQL, SQL Server, MongoDB, Redis, Steam and GameDig checks plus a remote-browser HTTP check; every field of every type is listed in Monitor types.

HTTP(s), Keyword, Json Query and DNS monitors take assertions on top of their own condition: status code, header, body text and JSON checks, or DNS record checks, all of which must pass. See Monitor types β†’ Assertions.

Marmot's type set follows Uptime Kuma's, so a monitor you know from there behaves the same here (the check code is in many cases a direct port, see THIRD_PARTY_NOTICES.md).

Intervals, retries and timeouts

Field Default Meaning
interval 60 s Seconds between checks. The UI minimum is 20 s.
maxRetries 0 Failed checks tolerated before the monitor goes DOWN. While retrying the monitor is PENDING.
retryInterval 60 s Seconds between checks while PENDING (usually shorter than interval to confirm an outage quickly).
resendInterval 0 Re-send the DOWN notification every N consecutive DOWN beats. 0 notifies once per transition.
reminderBackoff none Spacing of those reminders: none (every resendInterval), linear or exponential (see below).
maxReminders 0 Stop reminding after this many reminders per incident. 0 means unlimited.
successThreshold 1 Consecutive successful checks a DOWN monitor needs before it is UP again (see below).
timeout 48 s Seconds before a check is aborted. 0 means 80 % of the interval.
degradedAfter empty Milliseconds. A successful check slower than this is DEGRADED instead of UP (see below). Empty or 0 turns it off.
upsideDown off Invert the result: a failed check counts as UP and a successful one as DOWN (useful for "this port must be closed").

The state machine (a port of Uptime Kuma's) is:

maintenance window active                  β†’ MAINTENANCE (no notifications)
check ok, response time > degradedAfter    β†’ DEGRADED
check ok                                   β†’ UP
check failed, retries left                 β†’ PENDING, next check after retryInterval
check failed, no retries                   β†’ DOWN

Only important beats (a change between UP, DEGRADED, DOWN and MAINTENANCE, or the first beat when it is DOWN) notify. PENDING never notifies, so maxRetries: 2 with retryInterval: 20 gives a flaky endpoint 40 seconds to recover before anyone is paged.

Degraded

A service that answers correctly but slowly is not healthy. Set Degraded after (degradedAfter, in ms) on an HTTP(s), keyword, JSON query, TCP port, ping, DNS or gRPC monitor and every successful check whose response time exceeds it is recorded as degraded (yellow in the heartbeat and uptime bars, a "Degraded" badge, "Degraded performance" on status pages). The heartbeat message says by how much the threshold was crossed. Other types ignore the setting.

  • Degraded counts as up for uptime; the statistics count degraded checks separately (extras.degraded in the buckets, degraded in GET /api/monitors/:id/stats, "N degraded checks" on the detail page) and include their response times in the average.
  • Transitions UP ↔ DEGRADED ↔ DOWN are important beats. A failed check while degraded goes through PENDING and retries like any other failure; when the retries end, the monitor's status before them (status.settledStatus) decides whether anything changed, so DEGRADED β†’ PENDING β†’ UP still announces the recovery.
  • Upside-down monitors ignore the threshold (their UP comes from a failed check). Maintenance overrides degraded like every other status.
  • Notifications carry an event: down, up (recovered from DOWN, also to DEGRADED), degraded (UP ↔ DEGRADED) and reminder (resendInterval). Channels receive down, up and reminder by default; degraded is opt-in per channel (Notifications).

When the self connectivity check is on and the worker itself loses its internet connection, checks of external targets are held as PENDING checker offline beats instead: no notification, no downtime, and the monitor keeps the status it had before the outage.

Recovery threshold and reminder backoff

maxRetries delays the first DOWN, but by default one successful check brings a monitor back UP, so a flapping service produces a storm of DOWN/UP alerts. Two settings calm it down (#147):

  • Recovery threshold (successThreshold, default 1). A DOWN monitor needs N successful checks in a row before it is UP again (a slow, DEGRADED success counts too and then completes the recovery as DEGRADED). The checks in between are PENDING beats labelled Recovering 1/3: <message>, polled at retryInterval like retries, and notify nobody. A failure during the streak puts the monitor straight back to DOWN, without a new retry round, a new DOWN notification or a new incident, and the count starts over. The beat that completes the streak is the DOWN β†’ UP transition: it sends the UP notification and resolves the incident. Maintenance ends a streak as it ends an outage. It only applies to monitors that are DOWN; a PENDING retry after an UP beat still recovers on the first success. With successThreshold: 3 and maxRetries: 1, the check sequence ok, fail, fail, ok, fail, ok, ok, fail, ok, ok, ok sends exactly one DOWN and one UP notification.
  • Reminder backoff (reminderBackoff, maxReminders). The engine still marks a reminder every resendInterval DOWN beats; the base interval is resendInterval Γ— interval. With linear reminder k waits k base intervals after the previous one (1Γ—, 2Γ—, 3Γ— …), with exponential it waits 2^(kβˆ’1) (1Γ—, 2Γ—, 4Γ— …), so with exponential reminders go out 1, 3, 7, 15 … base intervals after the DOWN. maxReminders stops them after that many, and acknowledging the incident stops them in any case. The default, none with no cap, keeps Uptime Kuma's fixed cadence.

Scheduling is handled by the worker process through BullMQ job schedulers (one per active monitor); the web process only writes the monitor and nudges the scheduler. Several worker replicas share the load and a monitor is never checked twice at once (Architecture).

The monitor page

/{org}/monitors lists every monitor with its live status, a bar of the last 100 heartbeats and the 24 h uptime; it updates over the WebSocket connection without reloading (see search, filters and bulk actions below). The detail page adds:

  • uptime for 24 h and 30 d, average and current response time, and the number of degraded checks in the last 24 hours;
  • a period selector (24 hours, 7, 14, 30 or 90 days) and a percentile selector (p50, p75, p90, p95, p99) driving a metrics row (uptime, the highest selected percentile, total, failed and degraded checks, last check time), the response-time chart (one line per selected percentile plus the average) and a checks chart (up / degraded / down per interval). The figures come from the stat rollups (minutely for a day, hourly up to 30 days, daily for 90 days), so switching periods is cheap; percentiles are estimated from per-bucket latency histograms and stay within one histogram bucket (about Β±19 %) of the exact value;
  • for HTTP and TCP monitors, the request timing of the latest check and a phase chart that follows the same period selector;
  • the list of important events (status changes with their message) and the Logs tab with every check;
  • the certificate panel for HTTPS targets (issuer, expiry; filled by the certificate job landing in the current release);
  • the notification channels the monitor alerts through;
  • its recent incidents (see below);
  • actions: Check now, Pause/Resume, Edit, Clone, Delete.

Statistics are kept as minutely (24 h), hourly (30 d) and daily (KEEP_DATA_PERIOD_DAYS, default one year) buckets, so a monitor's history survives the pruning of raw heartbeats after 24 hours.

Search, filters and bulk actions

(landing in the current release, #124)

The list has a search box (press / to focus it, or Search monitors in the command palette) and filter menus:

  • the search matches every word against the name, the URL or hostname and the description, ignoring case and accents; words of three letters or more also match the name loosely (chkapi finds "Checkout API"). It understands filter tokens too: status:down tag:prod (also is:paused, type:http, location:<name> or location:local; tag: matches a tag's name, its value or name:value);
  • Status (up, down, pending, degraded, maintenance, paused), Type, Tags, Channels (not shown to viewers, who cannot read channels) and Locations (shown once the organization has a probe location; This server is the local worker pool). Values of one filter are alternatives, different filters and search tokens must all match.

Search and filters live in the query string (/acme/monitors?q=api&status=down,degraded&tag=12), so a filtered view can be bookmarked and shared. The list stays live while filtered: a monitor that goes down appears under Status: Down as soon as its heartbeat arrives.

Members (anyone with monitor:update or monitor:delete) get a checkbox per row. Click to select, Shift-click to select a range, or use the keyboard: Space toggles the focused row, ↑/↓ move between rows and Shift with either extends the selection. The header checkbox selects every monitor shown. The selection only ever holds monitors that are shown: a monitor hidden by a filter (or deleted) drops out of it. On the selection:

  • Pause / Resume;
  • Check now (one recorded check per monitor, queued without waiting; each spends one of the organization's ON_DEMAND_CHECKS_PER_MINUTE, and paused, push and probe-checked monitors are skipped);
  • Tags β†’ Add tag / Remove tag (an added tag keeps the value a monitor already has for it);
  • Channels β†’ Attach channel / Detach channel;
  • Maintenance β†’ window: adds the monitors to an existing maintenance window (needs maintenance:update too);
  • Status page β†’ page β†’ group: adds the monitors to a group of a status page as monitor components (needs status-page:update too);
  • Delete (after a confirmation; monitor:delete).

Pause, resume, check now, delete, Add to maintenance window… and Add to status page group… of the selection are also in the command palette while the list is open (the last two open the menu to pick the window or the group). Viewers see the search and the filters but no checkboxes.

Table view

(landing in the current release, #276)

The List / Table switch next to the filters shows the same monitors as a table with the columns status, name, target, type, tags, last check, 24 h uptime, average latency (24 h) and last incident (when the newest incident started, and whether it is open, acknowledged or resolved). Click a column header (or focus it and press Enter) to sort by it, again to reverse the order; monitors without a value sort last. Columns hides or shows columns (the name always stays). The chosen view, sort and columns are remembered per user in the browser. Search, filters, selection (including Shift ranges and the keyboard, in the table's order) and bulk actions work exactly as in the list, and the table stays live.

Each bulk action is one request to POST /api/orgs/:orgId/monitors/bulk; the toast reports how many monitors changed, how many already were in the requested state and the first failure.

Request timing

HTTP(s), keyword and JSON query checks record where the response time went, in milliseconds:

Phase Measured from β†’ to
DNS lookup start of the connection β†’ the host name is resolved (empty for an IP address)
TCP connect resolved β†’ TCP connection established
TLS handshake connected β†’ TLS handshake done (empty for plain http://)
Waiting (TTFB) connection ready (or request sent on a reused connection) β†’ first response byte
Content transfer response headers β†’ last body byte

The phases describe the final response: after redirects, earlier hops count towards the response time but not towards a phase, and DNS, connect and TLS are empty when that request reused a connection. Through a proxy only TTFB and transfer are measured (TTFB then includes connecting to the proxy). TCP port checks record DNS and connect. Checks run by probe agents report the same phases. Without redirects the phases add up to the response time, give or take a millisecond of request setup.

The phases are stored on each heartbeat (heartbeats.timing) and shown as a waterfall for the latest check and in the Check now / Test result. Successful checks also feed a per-phase average into the statistics buckets, which the Timing phases chart stacks for the period chosen in the period selector (the stats API's series points carry them as timing), so the history outlives the 24-hour heartbeat retention. gRPC checks do not record timing yet.

Response log

The Logs tab of a monitor (all types except push, which has its own ping log) lists every check, newest first: status, time, HTTP status code, response time, trigger (Scheduled or Manual for Check now) and the message, with the number of failed assertions. Filters: status (or Failed = down and retrying), status code class (2xx … 5xx), trigger and time range (last hour to last 30 days, or all). On a monitor checked from several locations, the page's location filter narrows the log to one location's checks. Load more pages further back. A row opens a sheet with the request timing waterfall, the assertion results, the per-probe results of a Globalping check, the response headers and the response body.

What an HTTP(s), keyword or JSON query check keeps on its heartbeat:

Field Stored
statusCode every check that got a response
response.headers every check that got a response, about 8 KB at most (headersTruncated beyond)
response.body failed (down, retrying) and degraded checks only, the first 16 KB
assertions, timing as described in Check now and Test and above

On a monitor checked from several locations, the reporting location's own outcome decides whether the body is kept, so a failing location keeps its evidence while the quorum is still up.

Only response data is stored, never the request: cookies the target sets (Set-Cookie) are replaced by [redacted], and every credential the check sent (the Authorization header, custom header values, the basic-auth password, bearer token, OAuth client secret and a password in the URL) is replaced by [redacted] wherever the target echoes it in a header or the body. The rows follow the heartbeat retention: 24 hours, status changes for KEEP_DATA_PERIOD_DAYS. Viewers can read the log; it is scoped to the monitor's organization.

Check now and Test

  • Check now (detail page, or the command palette on a monitor's page) runs the monitor's check immediately instead of waiting for the next interval. The heartbeat is stored with trigger: manual and counts like any other beat: it feeds retries, status, statistics and notifications, and the heartbeat bar updates live. A dialog shows the result: status, message, response time, HTTP status code, certificate and any check-specific details (such as assertion results).
  • Test in the monitor form runs the configuration you are editing once, before saving, and shows the same result under the form. Nothing is stored.

Both run on the worker, never in the web process, with the monitor's timeout, its proxy and the outbound address guard exactly like scheduled checks; with MONITOR_DENY_PRIVATE_ADDRESSES on, private and link-local targets are refused. They need monitor:update (Test: monitor:create or monitor:update), so viewers cannot trigger checks, and share a budget of ON_DEMAND_CHECKS_PER_MINUTE (default 30) per organization. Push monitors are fed by your system and cannot be checked on demand; paused monitors must be resumed first. Group and manual monitors cannot be tested before saving.

Incidents

An outage is recorded as an incident (monitor-incidents), so a team can see who is on it and how long it took. This is the internal, on-call side of an outage; the public side is the status-page incident (Status pages), and one can be created from the other.

  • Opened by the engine on the transition to DOWN (from UP, PENDING, MAINTENANCE or a first DOWN beat), with the message of that beat as its cause. A monitor has at most one unresolved incident.
  • Acknowledged by a member: from the incident page, the API or the signed link at the end of DOWN messages. While an incident is acknowledged, resendInterval reminders stop (they also follow the monitor's reminder backoff); the acknowledgement is sent to the monitor's channels ([name] [πŸ‘€ Acknowledged] Acknowledged by Ada.).
  • Resolved automatically when the monitor recovers (UP), with the duration shown. A member can also resolve it by hand (the channels are told); it then stays resolved until the monitor recovers and fails again. Entering MAINTENANCE adds a note and keeps it open.
  • Every step is on the incident's timeline, with who did it and an optional note.

/{org}/incidents lists incidents with filters (status, monitor, period), each one's duration and who acknowledged or resolved it, and a summary of the period: count, unresolved, MTTA (mean time to acknowledge) and MTTR (mean time to resolve). The list and the incident page update live. Publish to status page creates a public status-page incident (status "investigating", the page's components that show the monitor marked as major outage) and links it; resolving the monitor incident does not post to the status page, so public updates stay deliberate.

Viewers see incidents (monitor-incident:read); members acknowledge and resolve them (monitor-incident:acknowledge, monitor-incident:resolve); publishing also needs status-page:update. Incidents go with their monitor when it is deleted. A monitor that is already DOWN when this feature is deployed gets its incident on its next transition to DOWN.

The acknowledge link (/ack/<token>) is signed with a key derived from PAYLOAD_SECRET, names one incident and expires after seven days. It opens a page with an Acknowledge button, so link previews in chat apps cannot acknowledge. Anyone holding the message can use it: a signed-in member is recorded by name, anyone else as "from a notification link".

Declared incidents

A declared incident (declared-incidents, Response in the sidebar) is the incident-response record a team runs, whether or not a monitor noticed: a title, a severity (critical, major, minor), a commander (a member of the organization), who declared it, a summary, its start (backdated if needed, or taken from the monitor incident it was declared from) and a timeline.

  • Status: open β†’ mitigated β†’ resolved β†’ closed, or canceled (false alarm) before it is resolved. Mitigated may be skipped and a resolved incident can be reopened; any other change is refused (409). Closed and canceled incidents are read-only.
  • Timeline: append-only. Declaring, severity, status and commander changes, linking or unlinking the public status-page incident, postmortem drafts and approval, closing and notes each add an entry with who did it.
  • Postmortem: a Markdown document started from a template (Summary, Impact, Timeline, Root cause, What went well, What went wrong, Action items), saved as a draft and approved. Closing needs an approved postmortem; an admin can close without one (declared-incident:skip-postmortem), which the timeline records. Editing an approved postmortem makes it a draft again.
  • Commander: assigning someone emails them (not when they assign themselves). Without activity the commander is reminded after 1 h (critical), 4 h (major) or 24 h (minor), then after twice as long each time (at most a week apart); any change re-arms the reminder. Channels that selected the incident event hear about declarations, status and commander changes and reminders (Notifications).
  • Public communication stays on the status page: a declared incident links to a status-page incident (statusPageIncident) instead of copying it, so what visitors read is still posted deliberately.

Viewers read declared incidents; members declare and run them; admins delete them and may close them without a postmortem. Every change is in the audit log. API: GET/POST /api/orgs/:orgId/declared-incidents, GET/PATCH/DELETE …/:id (title, severity, summary, commander, status, startedAt, statusPageIncident, postmortem, approvePostmortem, skipPostmortem), POST …/:id/notes { message }.

Check locations

A monitor is checked by the workers of the Marmot server (Local) unless you switch on probe locations under Check locations: then probe agents in those networks run it, and its heartbeats record the location. Group, manual, push, Steam and Globalping monitors always run on the server.

Several locations and the quorum

Pick more than one location (probe locations, plus Local if the server's workers should check too) to tell an outage of the service from a broken path between one checker and the service. Every location runs the monitor on its own schedule with its own retries, degraded threshold, recovery threshold and checker-offline holds; the monitor's status is a quorum of the locations' statuses, set with Quorum:

Quorum DOWN when … 3 locations
Any location one location is down 1 of 3
At least half (default) half of the locations (rounded up) 2 of 3
All locations every location is down 3 of 3

Before that the monitor is PENDING while down and retrying locations together reach the quorum, and DEGRADED while degraded and failing locations do. A monitor checked from three locations therefore stays UP when one location fails and goes DOWN when a second one does. Only the monitor's own transitions are status changes: they are the important events, open and resolve incidents and send notifications, whose message names the failing locations (Down at 2 of 3 locations: Berlin (timeout), Paris (ECONNREFUSED)). A location failing on its own is recorded on its heartbeats and in the per-location table, silently. Reminders (resendInterval) count the beats of every location, so the interval is multiplied by the number of locations.

The monitor page then shows a Locations table (each location's status, response time, last check and 24-hour / 30-day uptime), a Response time by location chart and a location filter for the heartbeat bar, the events and the chart. The uptime cards and the main chart are the monitor-wide figures, computed from the quorum status.

SSH tunnels

Internal databases and services that are only reachable through a bastion host can be monitored through an SSH tunnel instead of deploying a probe agent. Admins add tunnels under Settings β†’ SSH tunnels (ssh-tunnel:create|update|delete; members can pick them, ssh-tunnel:read):

Field Notes
Name Shown in the monitor form and in error messages
Host, port The bastion (port 22 by default)
Username The account the worker signs in with; give it port forwarding and nothing else
Authentication Password, or private key (OpenSSH, PEM or PuTTY format) with an optional passphrase
Host key fingerprint Required. SHA256:… as printed by ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Then pick the tunnel under SSH tunnel in a monitor of one of these types: HTTP(s), HTTP(s) - Keyword, HTTP(s) - Json Query, TCP Port, PostgreSQL, MySQL/MariaDB, Microsoft SQL Server, Redis and SMTP. Write the target as the bastion sees it: host names are resolved by the bastion (postgres://[email protected]/app, http://10.0.3.7:8080/health). TLS still verifies the target's own name.

  • Host key pinning. The worker verifies the bastion's host key against the fingerprint before it authenticates; a different key refuses the connection (host key mismatch: the server presented SHA256:…, expected SHA256:…). Fetch from host in the tunnel dialog reads the key the bastion presents; compare it with the bastion's own before using it. Test connection connects, verifies the key and signs in.
  • Credentials are encrypted at rest (AES-256-GCM, key derived from PAYLOAD_SECRET, like two-factor and single sign-on secrets) and are write-only: no API returns them, not even to admins (hasPassword, hasPrivateKey and hasPassphrase say whether one is set), the audit log redacts them, and the dialog keeps a stored value when its field is left empty. Rotating PAYLOAD_SECRET means re-entering them.
  • Connections are pooled: each worker keeps one SSH connection per tunnel, shared by every monitor using it, with a keepalive every 15 s (dropped after 3 unanswered). A dropped connection is re-opened by the next check; one unused for 5 minutes is closed; editing the tunnel replaces it. Each check forwards its connections with forwardOut (direct-tcpip) through short-lived listeners on 127.0.0.1 that only exist while the check runs.
  • Failures. When the bastion cannot be reached or refuses the worker, the dependent monitors fail with SSH tunnel <name> unreachable: <reason>, so a bastion outage is not mistaken for an outage of the targets. When the bastion is up but cannot reach the target, the message says so: … (SSH tunnel <name> could not reach db.internal:5432: …).
  • Limits. A monitor uses either a proxy or a tunnel, and tunnels are opened by the server's workers only: monitors with probe locations cannot use one. UDP (DNS, SNMP, RADIUS, the SQL Server Browser of named instances), ICMP and multi-host protocols (MongoDB replica sets, Kafka) are not tunnelled. With the outbound address guard on, the bastion's address is vetted; targets behind it are the bastion's network and are not. Request timing then measures the connect phase to the local listener; the time through the tunnel shows up in TTFB.

Groups

A Group monitor has no target of its own. To put other monitors inside a group, set their parent. Groups nest, and they can be placed on status pages like any other monitor. This is the easy way to publish "API: operational" over a dozen internal checks. Deleting a group detaches its children instead of deleting them.

Aggregation modes

The group's aggregation setting (aggregation, #144) decides how the children's last status becomes the group's status. In every mode:

  • paused children are ignored;
  • children in maintenance count as UP;
  • a child with no beat yet, or one that is retrying, counts as pending.
Mode The group is
worst (default) DOWN as soon as one child is DOWN, PENDING while a child is pending, DEGRADED when a child is degraded and none is worse, otherwise UP. Groups created before #144 use this mode.
quorum DOWN only when fewer than aggregationQuorum children are up. The quorum is a number of children, or a percentage of the active children when aggregationQuorumUnit is percent (rounded up). Degraded children count as up. When the quorum is met but some children are not up, the group is DEGRADED. When the pending children could still make up the quorum, the group is PENDING.
weighted Each child scores 1 when up, 0.5 when degraded and 0 when down, multiplied by its weight. The weights are normalised to sum to 1. A score of 100 % is UP, 0 % is DOWN, and anything between is DEGRADED. Pending children are left out of the score.
  • Weights: each child sets its own weight (groupWeight, default 1). Weights are relative, so 3 and 1 give a 75/25 split. A weight of 0 means the child has no say. If every weight is 0, the children count equally.
  • Example: a quorum group of five replicas with a quorum of 3 stays up (DEGRADED) with two replicas down, with the message 3/5 children up (quorum 3); 2/5 children down: api-3, api-5. With a third replica down, it is DOWN. The usual retries and notifications apply.
  • Group response time: aggregationLatency sets it. none (default) is the time the group check took. avg, max and min are the average, the slowest and the fastest last ping of the active children.
  • Status pages: a group placed on a status page shows the group's own status, so its component follows the same aggregation.

Push monitors

A push monitor is checked by your system: cron jobs, backup scripts, IoT devices, anything that can make an HTTP request. Create a monitor of type Push; Marmot generates a pushToken and shows the URL to call, with copy-ready curl, bash-wrapper and crontab snippets.

Pings are expected either every interval or on a cron schedule ("0 2 * * *" in Europe/Berlin, the organization's time zone by default), with a grace period for late pings. Jobs can report more than "I'm alive": /start when they begin (a run that does not finish within the grace period goes DOWN), /fail or their exit code when they end, /log for intermediate output, and the output itself as the request body (the first 10 000 bytes are kept in the ping log). The time from start to success is recorded as the heartbeat's ping, so the response-time chart shows how long the job took; an optional maximum run duration reports slow runs. maxRetries and upsideDown apply as usual.

# at the end of your job
curl -fsS "https://status.example.com/api/push/<token>?status=up&msg=OK&ping=12"
# or: report start, outcome and duration
curl -fsS "https://status.example.com/api/push/<token>/start"
/usr/local/bin/backup.sh; curl -fsS "https://status.example.com/api/push/<token>/$?"

See Integrations for the endpoint reference and Monitor types for the scheduling rules.

Ping-by-email

Systems that can send mail but not make HTTP requests (NAS appliances, backup software, cron with MAILTO, network devices) can ping a push monitor by email. Once the instance is configured, every push monitor receives mail at <push token>@<INBOUND_EMAIL_DOMAIN>, shown on the monitor page. A +tag after the token is ignored (<token>+nas1@…).

Mail arrives through the worker's SMTP listener (INBOUND_SMTP_PORT; make it the MX of the inbound domain, or put a relay in front of it) or through the inbound-parse webhook POST /api/inbound-email (INBOUND_EMAIL_WEBHOOK_SECRET):

Provider Setup
Postmark Inbound webhook URL https://marmot:<secret>@status.example.com/api/inbound-email (JSON).
Mailgun A route for the domain that forwards to https://status.example.com/api/inbound-email?secret=<secret> (parsed fields, or body-mime for the MIME endpoint).
SendGrid Inbound Parse Destination URL https://status.example.com/api/inbound-email?secret=<secret>, with or without "POST the raw, full MIME message".
Cloudflare Email Routing An Email Worker that posts message.raw with Content-Type: message/rfc822, X-Marmot-Secret: <secret> and X-Marmot-Recipient: message.to.
Anything else POST the raw message (message/rfc822), Postmark-style JSON or { "to", "subject", "text" } with the secret.

Each message is matched, ignoring case, against its subject and plain-text body using the monitor's Email pings keywords, in this order:

  1. the failure keyword records a failure (e.g. FAILED);
  2. the success keyword records a success;
  3. the start keyword starts a run, like /start;
  4. without a success keyword, any other message is a success, so "Backup OK" works with no setup at all;
  5. with one, a message that matches nothing is ignored (kept in the ping log as a log entry) or recorded as a failure, as configured.

The subject becomes the heartbeat message and the first 2 000 characters of the body the ping log entry (source email). Mail to unknown tokens, paused monitors or other domains is refused (550 over SMTP, dropped with accepted: 0 on the webhook), and mail over INBOUND_EMAIL_MAX_BYTES is refused without being parsed.

Public name

The optional Public name (publicName) is what status pages show instead of the monitor's friendly name, so prod-api-eu-west-1 /healthz can appear as "API". A status page component can override it again with its own name (see Status pages β†’ Components).

Monitors-as-code key

The optional key (key, in the admin sidebar as Monitors-as-code key) is the stable identifier the marmot CLI maps entries of a monitors file to: marmot monitors apply creates, updates and (with --prune) deletes the monitors that carry a key. Keys are unique per organization and use letters, digits, ., - and _. Monitors without a key are left alone by --prune. A clone gets no key, and an import drops keys that are already in use.

Pause, resume, clone, delete

  • Pause sets active: false: the scheduler is removed, no checks run, the monitor keeps its history and shows as paused in lists and on status pages (paused monitors are dropped from public pages). Resume re-creates the scheduler and the next check runs immediately.
  • Clone copies every setting, tags and notification channels included, into a new, paused monitor named "… (copy)" so you can adjust the target before it starts checking. The status cache starts empty and push monitors get a fresh token.
  • Delete removes the monitor, its heartbeats and its statistics, and detaches child monitors from a deleted group.

Notifications, maintenance and tags

  • Notifications: the channels attached to the monitor (monitors.notifications) are alerted on important beats. Pick them in the Notifications card of the monitor form: it lists the organization's active channels, with the channels flagged as default already switched on for a new monitor. A change takes effect from the next status change. The detail page lists the attached channels (read-only for viewers) with a Test button for admins. From the channel side, Monitors in a channel's menu on the Notifications page attaches or detaches it in bulk, and Apply to all existing monitors attaches a channel to every current monitor. Clone keeps the channels and export/import carries them. The API takes notifications: [id, …] on POST/PATCH /api/orgs/:orgId/monitors; a create without the key gets the default channels. See Notifications.
  • Certificate transparency alerts: monitors with a hostname can announce every new certificate issued for it (optionally its subdomains), flagging issuers outside an allow-list (ctAlerts, ctIncludeSubdomains, ctExpectedIssuers in the Advanced card). See Notifications.
  • Maintenance: a monitor inside an active maintenance window reports MAINTENANCE instead of DOWN and does not notify; see Maintenance.
  • Tags: coloured labels with optional values (env: prod), managed under Settings β†’ Tags, shown in the monitor list, on the detail page and on status pages with showTags on.
  • Proxies: HTTP-type monitors can send their requests through an HTTP(S) or SOCKS proxy (Settings β†’ Proxies); the organization's default proxy is preselected for new monitors.
  • SSH tunnels: HTTP-type, TCP port, SQL database, Redis and SMTP monitors can reach internal targets through a bastion; see SSH tunnels.
  • Docker: the docker type checks a container's state on a Docker host (Settings β†’ Docker hosts; local socket or tcp:///https://). DOCKER_SOCKET_ENABLED=false forbids socket hosts on shared instances. Details in Architecture.

OpenTelemetry export

(landing in the current release, #99) Each monitor's checks are pushed as OpenTelemetry metrics to the organization's default OTLP collector. The form's OpenTelemetry card turns this off for the monitor (otlpExport: false) or sends it to another collector (otlpCollector, a collector id of the same organization; null = the default). Collectors, metrics and backend examples: Integrations.

SLA targets and reports

SLA in the sidebar (g l) tracks uptime commitments per monitor (#156). Everyone in the organization sees it; admins (sla:manage) set targets, build reports and share them.

Targets. A target is a monitor, a percentage (for example 99.9), a measurement window (calendar month, calendar quarter or the rolling last 30 days, all in UTC), an uptime calculation and whether maintenance windows are excluded:

  • Time up (default): time not DOWN divided by the time whose state is known, replayed from the monitor's status changes;
  • Successful checks: successful checks divided by all checks (degraded and maintenance checks count as successful, pending ones as failed).

With Exclude maintenance windows the maintenance time (or the maintenance checks) leaves both sides of the ratio and the error budget. For each target the page shows the current period's attainment, the error budget (the downtime the target allows over the whole period, e.g. 43.2 minutes for 99.9 % of a 30-day month) and what is left of it, the burn rate (the error rate divided by the allowed one: above 1 the budget runs out before the period ends) and the monitor incidents that consumed it. A target is on track, at risk (below the target so far or burning faster than allowed) or missed (budget spent).

Reports. A report groups targets. Its page lists, for any range of up to 400 days (default: last month), each target's uptime, downtime, incident count, MTTR (mean time to recovery of the incidents resolved in the range), the budget left and whether the target was met, and downloads it as CSV or PDF. A report can be emailed weekly (Mondays, for the previous Monday–Sunday) or monthly (on the 1st, for the previous month) to a list of recipients, with the CSV and the PDF attached; emails go through the instance's SMTP settings (Configuration).

Share links. Admins can create a public, read-only link to a report over a fixed range (/sla/<token>, a print-friendly page with CSV and PDF downloads). The token is shown once and only its hash is stored; a link shows that report and nothing else, and revoking it is final.

Figures come from the daily statistics, so a report reaches back as far as KEEP_DATA_PERIOD_DAYS. API: GET /api/orgs/:orgId/sla/targets, GET /api/orgs/:orgId/sla/reports/:id?from=&to=&format=, GET|POST /api/orgs/:orgId/sla/reports/:id/shares, DELETE …/shares/:shareId; targets and reports themselves are the sla-targets and sla-reports collections of the REST API.

API

Monitors are managed through org-scoped route handlers that authenticate the Payload session and apply the same permissions as the UI (monitor:read for viewers, monitor:create|update|delete for members):

Method & path Purpose
GET /api/orgs/:orgId/monitors[?key=] List (filter by type, active state or key)
POST /api/orgs/:orgId/monitors Create (body validated by monitorFormSchema)
GET /api/orgs/:orgId/monitors/:id/heartbeats Latest heartbeats, newest first
GET /api/orgs/:orgId/monitors/:id/logs Response log (see below)
GET /api/orgs/:orgId/monitors/:id/logs/:hbId One check in full (headers, body, assertions)
PATCH / DELETE /api/orgs/:orgId/monitors/:id Update / delete
POST /api/orgs/:orgId/monitors/:id/pause Pause
POST /api/orgs/:orgId/monitors/:id/resume Resume
POST /api/orgs/:orgId/monitors/:id/clone Clone (returns the new, paused monitor)
POST /api/orgs/:orgId/monitors/:id/check Check now (see below)
POST /api/orgs/:orgId/monitors/bulk One action on many monitors (see below)
POST /api/orgs/:orgId/checks Test an unsaved configuration (see below)
GET /api/monitors/:id/stats?range=&percentile= Uptime, ping, percentiles and chart series (below)
GET /api/orgs/:orgId/monitor-incidents Incidents + MTTA/MTTR (see below)
GET /api/monitors (Payload REST) List with Payload's where/limit/sort syntax

GET /api/monitors/:id/stats takes range = 1d, 7d, 14d, 30d, 90d (chart periods) or 24h, 1y (default 24h, same window as 1d) and an optional percentile list (p50,p95; all five when omitted). It answers { range, granularity, uptime, avgPing, degraded, buckets, percentiles, checks, step, series }: percentiles holds p50–p99 over the window, checks the total, up, failed, degraded and maintenance counts, and series one point per step seconds (timestamp, up, down, degraded, maintenance, ping, pingMin, pingMax and the requested percentiles, null without pings).

The log list takes status (one or a comma-separated list: down,pending), statusCode (503 or a class such as 5xx), trigger (schedule | manual), from / to (ISO instants), location (a location id, or local for this server's workers), limit (default fifty, at most 200) and cursor, and answers { docs: [{ id, time, status, msg, ping, statusCode, trigger, important, assertions: { passed, failed } | null, location, locationStatus }], nextCursor }; pass nextCursor back as cursor for the next, older page (null on the last one). The detail adds duration, retries, timing, assertionResults, probes and response: { headers, headersTruncated, body, bodyTruncated }.

Incident routes (monitor-incident:* permissions; ids are strings on MongoDB and numbers on Postgres):

Method & path Purpose
GET /api/orgs/:orgId/monitor-incidents ?status=active|all|open|acknowledged|resolved, monitor, range=24h|7d|30d|90d|all (by start, default 30d), page, limit β†’ { docs, page, totalPages, totalDocs, stats }
GET /api/orgs/:orgId/monitor-incidents/:id One incident with its timeline
POST /api/orgs/:orgId/monitor-incidents/:id/acknowledge { note? }; 409 unless open
POST /api/orgs/:orgId/monitor-incidents/:id/resolve { note? }; 409 when already resolved
POST /api/orgs/:orgId/monitor-incidents/:id/publish { statusPageId, title?, message?, status?, impact? } β†’ { incident, statusPageIncident } (201); 409 when already published
POST /api/incident-ack { token } from the signed link; no session needed
GET /api/monitor-incidents (Payload REST, read only) Payload's where/limit/sort syntax

stats holds total, open, acknowledged, resolved, mtta and mttr (seconds, null without data) over the monitor and period filters. An incident reads { id, monitor { id, name }, status, cause, startedAt, acknowledgedAt, acknowledgedBy { id, name }, acknowledgedVia (dashboard | api | link), resolvedAt, resolvedBy, autoResolved, remindersSent, lastReminderAt, statusPageIncident, timeline[] { type, at, by, via, message } }; the realtime monitorIncident event carries the same object.

Requests from outside the browser must send the payload-token cookie or a JWT Authorization header (POST /api/users/login returns one) and an Origin matching NEXT_PUBLIC_SERVER_URL. Organization API keys give machine access to badges and metrics (Integrations).

Bulk actions

POST /api/orgs/:orgId/monitors/bulk runs one action on up to 500 monitors:

{ "ids": [12, 13, 14], "action": "addTags", "payload": { "tags": [{ "tag": 3, "value": "eu" }] } }
action payload Permission
pause, resume β€” monitor:update
check (queue a recorded check, no waiting) β€” monitor:update
addTags, removeTags { tags: [{ tag, value? }] } (≀ 50) monitor:update
addNotifications, removeNotifications { notifications: [id] } (≀ 50) monitor:update
delete β€” monitor:delete
addToMaintenance { maintenance: id } monitor:update and maintenance:update
addToStatusPageGroup { statusPage: id, group: groupId } monitor:update and status-page:update

groupId is the id of the row in the status page's groups.

The answer is 200 with one result per id (duplicates collapse) and a summary:

{
  "action": "pause",
  "results": [
    { "id": "12", "ok": true, "monitor": { "id": "12", "name": "API", "active": false, "…": "…" } },
    { "id": "13", "ok": true, "unchanged": true },
    { "id": "99", "ok": false, "error": "notFound", "message": "Monitor not found" }
  ],
  "summary": { "changed": 1, "unchanged": 1, "failed": 1 }
}

error is notFound (not a monitor of the organization you can see), notApplicable (check of a paused, push or probe-checked monitor), rateLimited (check over the organization's on-demand budget) or failed (the write was refused, with the reason in message). Changed monitors come back in the realtime list shape (monitor), and the same change reaches every open list over the WebSocket.

Every monitor is written through the same path as the single-monitor routes (collection access, the organization checks of tags and channels, plan limits, scheduler sync, realtime events) as its own operation, so a failure affects only its id, on Postgres and on MongoDB alike. Everything that can be checked up front is checked before anything changes: the permission, the body, tags and channels of another organization (400) and the API key budget (an API key spends one write per monitor and may name at most API_KEY_WRITE_RATE_LIMIT monitors). Each changed monitor gets its own audit log row (monitor.paused, monitor.resumed, monitor.updated, monitor.deleted) with metadata.bulk = { action, count }; monitors that were already in the requested state get none.

addToMaintenance and addToStatusPageGroup leave the monitors untouched and write the maintenance window or the status page instead: they read it as the caller (404 when it is not one of the organization's, or the group does not exist), then update it once through the same path as its own form, so its access rules, organization checks, scheduling and audit row (maintenance.updated / status_page.updated, with metadata.bulk) apply. Monitors already in the window or the group (as a monitor component) are unchanged; the others are appended. If that single write is refused, every monitor it would have added fails with the reason.

On-demand checks

POST /api/orgs/:orgId/monitors/:id/check enqueues a check of the monitor on the worker (concurrent requests for the same monitor share one job) and waits up to the monitor's timeout for the result:

{
  "status": "down",
  "ok": false,
  "msg": "500 - Internal Server Error",
  "ping": 41,
  "statusCode": 500,
  "startedAt": "2026-10-07T09:00:00.000Z",
  "elapsedMs": 45,
  "blocked": false,
  "maintenance": false,
  "tls": null,
  "timing": { "dns": null, "connect": 0.4, "tls": null, "ttfb": 38.2, "transfer": 0.6 },
  "assertions": [
    {
      "kind": "status",
      "target": null,
      "comparator": "in",
      "expected": "200-299",
      "actual": "500",
      "passed": false,
      "legacy": true
    }
  ],
  "details": {},
  "recorded": true,
  "heartbeat": {
    "id": "123",
    "status": "down",
    "time": "2026-10-07T09:00:00.045Z",
    "important": true
  }
}

status is the beat's status after upsideDown, the degraded threshold and retries (a failing check with retries left is pending); assertions lists the per-assertion results of HTTP and DNS monitors (the same shape as heartbeats.assertions), timing the request timing phases (null for types that measure none) and details carries other check-specific fields. Query parameters: wait=false answers 202 { "jobId", "monitorId", "status": "queued" } at once (the heartbeat arrives over realtime), record=false runs the check without storing a heartbeat or changing the monitor's state.

POST /api/orgs/:orgId/checks takes the same body as creating a monitor and returns the same result shape with recorded: false; nothing is stored. Errors: 400 invalid body, untestable type (push, group, manual) or a target the address guard refuses; 403 without the permission; 404 unknown monitor; 409 paused or push monitor; 429 organization budget used up (Retry-After); 504 no result in time (is the worker running?). Organization API keys will be accepted here once API keys can call the management API (#115); until then use a session as described above.

⚠️ **GitHub.com Fallback** ⚠️