Path Monitor - rogers-group-inc/polaris GitHub Wiki

Path Monitor

A path check tests whether a web page, a service or an address can be reached from the machines that run the Polaris Agent or from the Polaris server itself. Each source runs the check on its own schedule and reports:

whether it passed, and how long it took (split into DNS, connect, TLS and time-to-first-byte for web checks);
for web checks, the HTTP status, whether the body matched, the TLS certificate's issuer and expiry, and a fingerprint of the response;
a traceroute of the path the traffic took, with each hop matched to the device Polaris monitors at that address.

Checks live on the Path Monitor page (sidebar, under Application Map). Results appear on each host's Paths tab and on the check's Results view, where the Polaris server has its own row.

A check measures the path from a host. It never changes that host's own Up / Down status — a laptop that cannot reach the intranet is not a laptop that is down (business rule 85).

What you need

The Polaris Agent 0.21.0 or later on the hosts that should run the check. Older agents are listed on the check's Results view with upgrade agent; they run nothing until upgraded. A check that runs only from the Polaris server needs no agent anywhere.
The Path Monitor permission. Read-Only shows the list and the results; Read-Write creates, edits and deletes checks. It is its own permission because a check tells every matching agent to send traffic to a destination on a schedule.
To run a check from the Polaris server, Read-Write on Network Discovery as well. The server probes from its own network, which is often one no agent host can reach, so pointing it at a target is treated like a Discovery sweep. Without it the Run from this Polaris server toggle is greyed out. You can still turn the server off on an existing check (it then runs from agent hosts), or rename it.

No extra privilege is needed on the hosts. HTTP, HTTPS, TCP, ping and traceroute all run under the agent's normal (unprivileged) service account. The one exception is below under Troubleshooting.

Creating a check

Click + Add Monitor. The Add Path Monitor dialog walks through four steps: General → Expectations → Traceroute → Sources. The bar across the top shows where you are, and the line between steps fills in as you move past each one. Use Next and Back (or the → / ← keys) to move. Next checks the step you are leaving, so a check with no name cannot leave General. Create appears on the last step. You can click any step you have already visited. When you edit a check, every step is open and Save Changes shows on all of them.

General

Field	What it means
Kind	HTTPS / HTTP send one `GET` request. TCP opens a connection to a port. ICMP sends one ping.
Target	A full URL for HTTP / HTTPS (`https://intranet.example/health`), `host:port` for TCP (`db01.example:5432`), a host or IP address for ICMP.
Every (minutes)	How often each host runs it: 1 to 60 minutes.
Timeout (ms)	How long a run may take before it fails: 500 to 30 000 ms, and at most half the interval.

Some targets are refused: loopback, link-local (including cloud metadata addresses), multicast, IPv6 addresses, URLs with a user name or password in them, and the Polaris server itself. Private (RFC 1918) addresses are allowed. The agent checks the address again after it resolves the name.

For HTTP and HTTPS, the Request section sets how the request is sent:

Field	What it means
Method	GET (default) or HEAD, which fetches the headers only, so there is no body to check. A check never sends anything that changes data (no PUT, POST or DELETE).
Host header	Sent instead of the URL's host name, and used for the TLS certificate check. Point the URL at one server's address and still ask for the site by name, for example to test each server behind a load balancer.
Follow redirects	Off by default, so a 302 is judged as a 302. On follows up to 5 redirects and judges the final page. Each new address is checked like the first.
Authentication	None, or an HTTP credential (Bearer token, Basic or Digest) from Server Settings → Credentials. See below.

A check using Method HEAD, a Host header, Follow redirects or a NOT body check needs Polaris Agent 0.23.0 or later on agent hosts; older agents skip it and show upgrade agent until they are upgraded.

Authentication runs only from the Polaris server. A check that authenticates is never sent to an agent, so the password or token stays on the Polaris server. Choosing a credential turns on Run from this Polaris server and locks it on; setting Authentication back to None hands the toggle back. Anyone with at least Read-Only on Credentials can pick any credential and test with it; changing the credential itself still needs the usual Credentials permissions. A credential a check uses cannot be deleted until the check stops using it. Authentication and the Host header are sent only to the original site: a redirect that leaves it gets neither. Basic and Bearer over plain http:// send the secret unencrypted, and the dialog warns you.

Expectations (HTTP / HTTPS only)

Accepted status codes — codes and ranges, comma-separated, e.g. 200,204,300-399. Blank means any 2xx. Unless the check follows redirects, a 302 is judged as a 302.
Body must contain / equal exactly / match a regular expression, or NOT contain / equal / match one: use a NOT option to fail the check when an error or maintenance page comes back. Checked in the first 64 KB. Regular expressions — on an agent or on the Polaris server — do not support lookahead, lookbehind or backreferences (rule 116).
Verify the TLS certificate — on by default. Off accepts any certificate; the check still reports the certificate's issuer and expiry.
Keep a body excerpt on every run — see What is stored below.

TCP and ICMP checks pass when the connection (or the ping reply) arrives within the timeout.

Test from this Polaris server (at the bottom of the step) sends the request once, with what you have filled in so far, and shows what came back: whether it passes the expectations, the status code, the timings, the TLS certificate, the response headers and the body. Nothing is saved. Two buttons turn the answer into the expectation: Accept only <code> sets Accepted status codes to the code that came back, and Require the selected text makes the body check contain whatever you have selected in the response. The test runs from the Polaris server, so an agent host on another network can get a different answer. It needs the same Network Discovery permission as running a check from the server, and you can run up to 10 tests a minute.

Traceroute

On by default. A traceroute runs on the first run, every N runs after that (default 5), and immediately whenever a run fails after a passing one, so a failure always has a fresh path to look at. Max hops (default 30) and Probes per hop (default 3) set how far and how thoroughly it looks.

The Polaris server traces with the system traceroute (or tracepath). If neither is installed, its traces come back empty with a note saying so (see Installation).

Sources

Where the check runs from. One toggle, Run from this Polaris server, picks the source:

On: the server that hosts Polaris runs the check itself, whether it is installed on Linux or in a container, with no agent needed. This is the one vantage point every install has. The agent-host filter is hidden.
Off: agent hosts run it, always limited to hosts with an active Polaris Agent. Either:
- tick All agent hosts: every host with an active agent runs it, including hosts added later; or
- find and pick hosts: build a device filter with the same condition builder the automation wizard uses (for example Tag is applied Camera Station). The list underneath shows every agent host it finds. Tick the hosts that should run the check; only ticked hosts run it. The Select all / none box at the top of the list ticks or clears every match, including matches past the first 100 rows shown.

The filter only finds hosts. It is saved so the check reopens with it, but a host that matches it later does not join the check by itself: open the check and tick it. A ticked host keeps running the check even if it stops matching the filter (it is marked outside filter).

A host runs at most 20 checks. If it matches more, it runs the oldest 20, and Polaris writes a path_check.agent_over_cap event naming it.

Reading the results

The list's Result column counts how many sources' latest run is in each state, for example 3 OK · 1 Unexpected response · 2 Fail (a state with no sources is left out):

Result	Means
OK	the source's latest run passed
Unexpected response	a source got an answer from the target, just not the expected one: a status code outside Accepted status codes, or a body that failed its check
Fail	a source got no answer at all (timeout, refused connection, DNS or TLS failure)

The Results view and a host's Paths tab use the same three words for each source.

Results (from a check's row menu) lists every source that runs it, with the latest result, latency, HTTP status, resolved address, hop count and the last error. It refreshes on the check's interval while it is open. Click a host to open its Paths tab. The Polaris server row (marked server) comes first. Click it and choose Show charts and path to see the same charts, latest result and path graph as a host's Paths tab, below the table.

The host's Paths tab shows:

a table of every check the host runs — click one to see it below;
Latency over time. Failed runs show as red dots on the baseline. Use the chips above the chart to add the DNS, Connect, TLS and TTFB lines. If an automation sets a latency SLA on this check, its threshold is shaded on the chart;
Availability — the share of runs that passed. A gap means the agent did not report (for example, the host was off), not that the check failed;
HTTP status — one cell per run, green for a pass and red for a failure;
Latest result — including the body fingerprint and size, the TLS issuer and how many days are left on the certificate, and the error text;
Path — the traceroute, as a graph and a table. Pick an earlier trace from the list to compare. Hops that changed since the previous trace are marked in the table. A hop that is a device Polaris monitors links to it, with its status at the time of the trace, and shows the subnet it sits in.

Reading the path graph

The graph combines the host's last ten traces. It reads left to right: this host, then one column per hop, then the destination.

Branches are routes that changed. When recent traces took different routers at the same hop, the graph splits there and joins again where the routes meet. Thicker links were taken by more traces.
The selected trace is drawn solid on top; the other routes are faded. Each of its links is coloured by the latency that hop adds: green under 10 ms, amber for 10–50 ms or when some probes got no reply, red over 50 ms. The red link is usually the one to look at.
Circles carry the hop number. A hop Polaris monitors is filled with its status colour — click it to open the device. A dashed * is a router that did not reply; that is common and does not mean traffic stopped there.
A dashed red link with a cross into the destination means that trace never reached it.

Hover a circle for its address, reverse DNS, round-trip times, lost probes and how many of the recent traces passed through it.

Exporting the path graph

The Export button at the right of the Path heading opens a menu:

Copy screenshot: the whole graph and its legend as an image on the clipboard, even the part the panel has scrolled out of view. Copying an image needs HTTPS.
Export PDF…: the graph on paper, in the light colours whatever theme you use. A header names the source, the destination, the check type and the trace shown, and a key in a box at the bottom left explains the link and status colours. Choose the paper, the orientation and the smallest text size allowed (7 pt by default). A path too long for one sheet at that size is split across several. The source and destination print as S and D, because the PDF fonts have no house or target symbol.
Export to Visio (.vsdx): one Visio page, every circle, label and link a separate editable shape. Links are drawn lines, not glued connectors. The same header sits above the graph and the same key box at its bottom left, as on the PDF.

The export shows the trace selected in the list, with the routes the other recent traces took as faded branches, as on screen. The same button appears on the Polaris server's results under Path Monitor → Results. The Device Map and the Application Map have the same menu (Device Map, Application Map).

Alerting — setting an SLA

A check has no threshold of its own. To be alerted, create an automation and pick Path Monitor as its trigger type (details). It watches the agent hosts' results and, when Include the Polaris server's own runs is ticked, the server's too. Its conditions are the path-check metrics:

Metric	Meaning
Path latency	ms for a run. A failed run has no latency.
Path failure rate	% of runs that failed over the History window, like packet loss
Path check result	Reachable / Unreachable for each run
Path HTTP status	the status code returned
Traceroute hop count	hops in the latest traces
TLS certificate days remaining	days until the target's certificate expires

Pick the check in the condition's Check picker, or leave it blank to watch every check each host runs (one alert per check). The automation only ever watches hosts with an active Polaris Agent; the Devices step narrows those. Holds, severity bands, resets, maintenance windows and dependency suppression all work as for any other automation. The Polaris server is the exception: no device filter, maintenance window or dependency applies to it.

The alert email shows the path. A Path Monitor alert's email carries the check's path graph, drawn as on the Paths tab: the recent traceroutes folded into one view, each link coloured by the latency it adds on the newest trace, with a ✕ where the destination was not reached. A grouped alert shows up to four graphs — one per device when it lists several devices, one per automation when it lists one device — worst first. Nothing is drawn while traceroute is off for the check or before its first trace. An email layout of your own needs the {path.graph} variable to include it.

For route changes, set the Path Monitor trigger to fire when a check's route changes (traceroute). Polaris writes this event (path_check.path_changed) at most once every 10 minutes per host and check, naming the host. A route change seen from the Polaris server is written too, naming the check instead. It fires the automation when the server is ticked.

What is stored

Kept	For how long
Each run's result, timings, status, fingerprint (SHA-256) and size of the body	the Path checks row of the Retention card (Server Settings → Maintenance), default 7 days raw / 30 days hourly / 365 days daily
Up to 4 KB of the response body — only when the run failed, or on every run when Keep a body excerpt is on	the same
Traceroutes	the Path traceroutes row, default 30 days

Response bodies can contain sensitive data. Leave Keep a body excerpt off unless you need it.

Deleting a check stops the agents running it on their next config refresh (within a few minutes), and the Polaris server at once. Its past results are not deleted immediately; they age out on the retention schedule.

Troubleshooting

Symptom	Look at
A host never shows results	the agent version (0.21.0+), and whether the host appears on the check's Results view. A host that is not listed is not ticked on the Sources step (matching the filter is not enough)
The Polaris server row never shows results	that the check is enabled and Run from this Polaris server is on. Server-run checks start within a minute of saving and run on the web role, so on a split install look at that service's log
The server's traces have no hops and a note about a missing tool	install `traceroute` on the Polaris host and restart Polaris. The check itself is unaffected
An ICMP check from the server fails with `icmp unsupported on this server`	the system `ping` could not run for the Polaris service account. The same `ping` serves Polaris's own ICMP monitoring, so fix it there
ICMP check fails with `icmp unsupported on this host (ping_group_range)`	Linux only. See Polaris Agent → Troubleshooting for the one-line fix. HTTP, TCP and traceroute are unaffected
Every hop after the first shows `* * *`	the network drops the ICMP replies traceroute relies on. The check result is unaffected
A few middle hops show `* * *` (often the carrier's core, several in a row) but `tracert` or `traceroute` on the same host lists them	upgrade the agent to 0.23.1 or later, which sends one probe at a time, as `tracert` does. Carrier (MPLS) core routers drop most of a burst of replies, and an agent before 0.23.1 sends its probes as a burst. A trace can take a little longer when hops stay silent
An agent host on Linux never reaches the destination, but the Windows hosts do	Linux agents trace with UDP probes and learn they arrived from the destination's port unreachable reply; a target whose firewall drops UDP never sends it. The hops up to it are still correct
An HTTPS check fails with a certificate error	the host does not trust the target's certificate. Fix the certificate or its chain, or turn off Verify the TLS certificate
A 302 fails the check	redirects are not followed unless you tick Follow redirects (General step). Or point the check at the final URL, or accept `300-399`
An authenticating check shows no agent hosts	by design: a check with a credential runs only from the Polaris server, so the credential never reaches an agent
A check says upgrade agent on every host	it uses HEAD, a Host header, Follow redirects or a NOT body check, which need agent 0.23.0