Dependency Suppression - rogers-group-inc/polaris GitHub Wiki

Dependency suppression

When a FortiGate goes dark, everything behind it stops answering too. Without suppression that is one outage reported as two hundred — and the two hundred bury the one that matters.

Polaris builds a dependency graph, and a device whose upstream is confirmed down is not accused.

What you see

A suppressed device shows a slate-blue "Dep. Down" pill. Suppression outranks the monitor-state label — including the device's own probe-down — and the device's own state moves to the tooltip. Only a maintenance window or a running dependency test outranks it.

The same colour is used consistently: Device Map pins, cluster icons, topology nodes and the mobile asset detail. A cluster counts suppressed children as dep-down, while a non-suppressed down or degraded child still rolls the cluster up red or amber.

On charts, a failure the upstream explains is drawn grey rather than red (rule 38b). Same dive, no accusation. It is deliberately not a band — that vocabulary belongs to maintenance.

The graph

Edges live on the asset and come from these sources:

Source	Built by
`computed`	rebuilt at the end of every FortiManager / FortiGate discovery cycle, from interface adjacency, LLDP, mesh and controller signals
`endpoint`	a fleet-wide pass giving each endpoint one parent
`vcenter`	VM → ESXi host placement
`unraid` / `truenas`	VM or container (App) → the Unraid / TrueNAS host it runs on
`proxmox`	VM or LXC container → the Proxmox VE node it runs on. When a guest migrates, its edge moves to the new node on the next read
`override`	operator-managed, never touched by recompute

Layer assignment is physical-first: a breadth-first walk from any FortiGate (layer 1), using interface, LLDP and mesh adjacency as the primary signal, with controller edges as a fallback for assets the physical pass did not reach.

Endpoints get exactly one parent

The tree covers endpoints too, so a server or camera behind a dead gate reads "Dep. Down" alongside the switches behind the same gate rather than alerting as plain Down.

An endpoint's parent is the most specific of, in order:

its last-seen switch port
its last-seen wireless AP
the freshest resolving FortiGate sighting
the gate that owns its address per IPAM

Never a union (rule 45). All-down semantics exist for redundancy, but a switch and its gate are in series — listing both would let a dead switch under a healthy gate keep the endpoint alerting.

The fourth tier is an inference, not an observation (rule 55). It fires only for endpoints the three observed tiers left unplaced, so it can only ever add suppression, never move an edge: the failure mode is a missed alert, never a false one. It is gated on the endpoint's address claim being current, because a recycled DHCP address would otherwise suppress a departed device behind whoever serves that range now.

Each parent row in the asset's General tab carries a detectedVia tag — "last-seen switch port" / "last-seen access point" / "last-seen firewall" / "owning firewall (from IPAM)" for an endpoint, and controller, interface, LLDP, wireless mesh, hypervisor placement or operator override for the rest — because an edge that silences alerts should say which signal put it there. An endpoint with no resolvable upstream gets an explicit "dependency suppression can't apply" empty state rather than a missing block.

Multi-parent: all-down

A device with N effective parents suppresses only when every parent is down or itself suppressed. A switch with redundant uplinks keeps alerting while either uplink is healthy.

An empty parent set never suppresses. That is why a broken parent lookup is invisible-but-total: a child that resolves no parent does not fail loudly, it just never suppresses — so a broken lookup shows up as "a FortiGate in maintenance leaves its switches reading Down" rather than as an error anywhere.

Overrides

PUT /assets/:id/dependencies/override (assets:write), over the API — there is no editor for it in the UI. If any override row exists for an asset, those are the effective parents and computed rows are ignored. An empty override set is an explicit "no parents" pin — the asset opts out entirely. DELETE on the same path clears the overrides and hands the asset back to the computed graph.

Cycles are rejected.

Entering and leaving are asymmetric

This is the part most worth knowing (rule 38a).

Entering suppression needs every effective parent confirmed down. Warning and recovering flapping must never drag a healthy subtree in.

Leaving needs the parent genuinely back. Only these states release:

up · unknown · passive

Everything else holds — down, recovering and warning all hold the subtree.

unknown and passive release because neither is a claim about reachability, and gating on them would strand a subtree with nothing able to clear it. That argument does not apply to warning: it is the one state in which the parent has just missed a poll — the least plausible moment to call it recovered — and it can strand nothing, being transient by construction.

Releasing on warning would let a parent flapping down → recovering → recovering → warning put its whole subtree back on the air mid-outage one reconciler tick later, every child re-alerting as plain Down — precisely the storm suppression exists to prevent.

A gate coming back does not mean the switch under it has, so each layer still needs its own parent genuinely back. But a suppressed device keeps answering its own probes at the normal interval, and its own count keeps draining while the parent recovers (rule 38(c)). So when the gate reaches up, every device below it whose own count has already reached up leaves Dep. Down in the same moment, all the way down the chain. A device that is still recovering is released but keeps its own children held until it reaches up itself.

What suppression actually does


Heavy cadences	paused
Response-time probe	still runs, at the normal interval — the device may answer over a redundant path, and its own count is how it comes back with its parent
Probe failures	stamped as dependency-explained, rendered grey
Alerts	the device is excluded from firing (rule 37) — unless a down automation opted to speak for it, below
Live alerts	retired when the upstream is genuinely down. Behind a parent in maintenance, a device-down alert is cleared and every other alert is kept (paused) (rule 16(a))

A suppressed device is still probed, at the same interval as always. That is deliberate: a device with a redundant path may well answer, and when the upstream recovers the device behind it recovers at the same time. Its own count is already caught up when the upstream reaches up, so it doesn't sit at Recovering afterwards.

One automation may speak for it anyway

Silence is the right default for the NOC, and the wrong one for the people who only care about one device. A plant operator subscribed to a PLC's down automation hears nothing when the switch above the PLC dies, because the PLC is Dep. Down and the switch is somebody else's alert.

A monitor status is down automation can opt out of the silence with the Dependency-Down Bypass row on its Actions step, under the In-app Alert card (rule 78). With it on:

the automation still raises its alert the moment the device turns Dep. Down — the upstream's confirmed verdict is the evidence, so it does not wait for the device's own missed-poll count;
the alert says DEPENDENCY DOWN in its subject, headline and message, and names the upstream device that is down — and, when that device is itself Dep. Down under something further up, the device that is actually down (a suppressed switch's FortiGate);
its email draws the dependency path under the alert's details: the device that is actually down on the left, this device on the right, every device in the location box its a: / b: / f: / r: / jb: codes put it in on the Device Map (a device with no codes sits in a plain box named after its Location), and the port names on each link where LLDP knows them. At most four devices are drawn: a longer chain shows the two at each end with a +N more gap between. The plain-text email gets the same chain on one line. The diagram is the {dependency.path} token, which is in the default email — an automation whose email you have customized may not have it, so add it where you want the diagram. The reset email (the one saying the alert is over) draws the same devices as they are now, under the heading Dependency path now: each one is marked Up, Recovering, Missed poll, Down, Dep. Down or In maintenance, read when the email is sent. It never assumes the whole chain recovered: if the device that was down is still recovering, the picture says so;
it is one notification: reminders and escalation tiers wait, as they do for every suppressed device, until the upstream is back;
it is acknowledged with the device that is actually down (rule 78a): acknowledging that device's own Down alert acknowledges every dependency-down alert naming it, with the same person and note, and one raised afterwards arrives already acknowledged (it is still sent). Hover its ack pill on the Active Alerts widget to read the note, marked as inherited from the root cause's alert. Acknowledging a dependency-down alert acknowledges nothing upstream;
a live plain Down alert on a device that then turns Dep. Down is ended and raised again in the dependency flavour, so the operators hear which device is responsible — and on the way back, a dependency-down alert whose upstream recovered while the device stayed dark is raised again as the device's own outage. Neither handoff sends a "resolved" message; nothing recovered;
a maintenance window still silences it — announced downtime is not an outage to report.

Every other automation keeps ignoring a suppressed device, and the toggle is refused anywhere but on a sole monitor status is down condition.

Timing

Two paths keep it current:

Event-driven — a confirmed up/down transition propagates immediately. Warning and recovering churn logs an Event but does not propagate.
A 60-second reconciler — the source of truth, catching anything the hook missed. It runs in layer order, so a parent's effective state is settled before its children evaluate; otherwise multi-tier suppression could oscillate.

Transitions write monitor.dependency_suppressed and monitor.dependency_resumed Events.

Rules that keep the graph honest

A retired device is nobody's parent. decommissioned and disabled infra assets are dropped from the graph and kept in scope, so their own rows are deleted rather than frozen.

Why that is not cosmetic: an unmonitored parent is transparent — the walk recurses to grandparents and returns "ok" when there are none. A firewall is layer 1 with no parents, so a retired gate is a permanent ok vote that vetoes suppression for every child still bound to it, and it can never go down again to lift the veto — a switch behind a replaced gate would sit at plain Down instead of Dep. Down.

A stale LLDP row is not good enough to draw an edge from. An LLDP neighbour is kept for 48 hours so the System tab's Neighbor column does not flap on a missed advertisement — but the dependency recompute filters those rows to roughly 6 hours first.

The bound is tight because the failure directions are asymmetric: a spurious parent breaks suppression outright under all-down semantics (an extra parent that is up keeps the child alerting as plain Down when its real upstream dies), while a missing LLDP parent is usually covered by the same asset's controller and interface edges.

The gate's own managed inventory bounds the graph. Discovery asks each FortiGate for the switches and APs it manages and stamps the answer on the gate; any adjacency that roster contradicts is dropped, in both directions.

It is tri-state: absent means unknown and constrains nothing; an empty list means manages none; a populated list constrains. The stamp is written only when the roster read answered — blur that distinction and a failed read reads as "manages nothing" and a gate silently loses its subtree.

Membership can only reject an edge, never assert one.

Wireless attachments override the controller signal. A mesh-leaf AP depends on its root AP, not on its controller. A FortiLink switch bridged behind a FortiAP depends on the AP.

Parent resolution never matches a FortiGate by hostname. It goes through the shared parent-key resolver, because controllerFortigate holds FortiManager's device name and "no parent" is a legitimate state — so the mismatch fails silently, in exactly the invisible-but-total way described above.

A hostname shared by several switches or access points is settled by where the device sits, or not at all (rule 91). FortiLink fleets often name switches per site, so "IDF-1" sits behind every gate, while Last Seen Switch, Last Seen AP and an access point's uplink switch are all recorded by name. A shared name is resolved by the switch whose MAC table holds the device, then by the FortiGate that last saw it (an access point's own controller), then by the FortiGate that owns its network. Nothing decisive places nothing: the device hangs off the FortiGate that saw it instead, so it is still at the right site. In the tree, rows that share a name carry their serial number beside it.

Seeing the tree

The asset's General tab renders the dependency tree for every asset type. An endpoint sees its own dependency as the parent row above it — that is the only surface where an endpoint edge is displayed.

Each row carries two separate signals. The pip at the left is the device's monitor state (▲ up, ▼ down, ● Dep. Down, and so on). A coloured dot after the name means the device has an active alert — the same dot as on the Assets list, coloured by its most severe alert. It pulses while any of those alerts is unacknowledged and sits still once all of them are acknowledged; hover it for the count. A device can be up and still alerting (high CPU, a down port), which is exactly what the dot is there to show.

The downward view (children and grandchildren) is deliberately infra-only and capped. Endpoints are real children of the switch, AP or gate that last saw them, but listing them buries the infra chain the tree exists to show — and a site gate is the last-seen device for the entire site.

Testing it

Pick a live FortiGate, drive its monitorStatus to down, and wait one reconciler tick (≤ 60 s). The child switches and APs should flip to Dep. Down and emit monitor.dependency_suppressed.

There is also an outage simulation for testing dependency behaviour without a real outage: Simulate Down, left of the Status pill on the System tab of a monitored firewall, switch or access point. It asks for a duration (1–240 minutes, 30 by default); for that long the device counts as down for its children, which turn Dep. Down exactly as for a real failure, while its own probes keep running. Its pill reads Dependency Test, the dependency tree marks it, and it clears itself at the deadline — or at once with Clear Dep. Test, after which children resume within about a minute.

It is gated at assetMonitorSettings:fullwrite — the admin-only level — because it stamps a field that can mask a real outage.