Edge Ban Detector - digitalspace/eagle-demi GitHub Wiki

Edge ban detector

The edgeBan timer in the DEMI API finds addresses that pull many documents through Front Door with no sign of a browser, and keeps the banauto rule's address list in step with the bans it holds. Code: src/scripts/edge-ban.js, src/repositories/edge-bans.js, timer in api/index.js. The Front Door rules, the placeholder and the manual list are in the eagle-dev-guides wiki page Front-Door-Ban-Ladder.

What one run does

The timer runs hourly (EDGE_BAN_SCHEDULE, default 0 0 * * * *). A run:

  1. Reads the Front Door access log in eagle-logs-<env> for the last 24 hours, as demi-identity-<env>.
  2. Groups requests by socket address: IPv4 as /32, IPv6 as /64.
  3. Counts distinct documents per address. A document is a path matching documents/<id>/download or /fetch. Range requests for one document count once.
  4. Counts browser evidence per address: SPA loads (spafallback matched), analytics beacons (usagealias or /api/analytics) and calls to /demi-search/search.
  5. Judges each address with more than documentFloor documents.
  6. Records bans in the edgeBans container, then rebuilds the list from unexpired bans.

An address's traffic before its last ban ended is not counted again.

Judge order

First match wins.

Verdict When
never ban Address is in the policy allow list
never ban Address is in a published crawler range (Google, Bing, OpenAI feeds)
never ban User agent holds a declared crawler name (built-in list plus policy crawlerAgents)
never ban Owner type is government, institution, mobile or relay
leave Browser evidence is above evidenceFloor
ban Owner type is datacenter or vpn
watch Anything else (residential, unknown): reported, never banned

Owner type comes from an RDAP lookup (rdap.arin.net) of the network name and registrant names, matched against word patterns. A lookup that fails gives unknown. A known row's owner is reused.

The crawler range feeds are cached in edgeBans for 24 hours. A failed fetch falls back to the cached copy at any age.

Escalation

A ban lasts 1 hour for the first offence, 4 hours for the second and 24 hours from the third on (banHours). The count comes from the address's row, which expires 7 days after its last write. An address quiet for 7 days starts at 1 hour again.

Guards

Guard Value
List entries IPv4 /32 and IPv6 /64 only. A wider value in a row is refused with a warning.
List length maxListLength 100. Over it, the latest-expiring bans are kept and a warning is logged.
Bans added per run maxNewBansPerRun 20. The rest are marked deferred and judged again next run.
RDAP lookups per run maxLookups 50. Past it, owner is unknown, so the address is only watched.
Crawler ranges With no cached or fresh crawler ranges, the run bans nothing.
Empty list The placeholder 192.0.2.1/32 is written.

Modes

EDGE_BAN_MODE sets what a run writes.

Mode Cosmos edgeBans banauto rule
off Not read or written Placeholder written. Needs no policy. Kill switch.
shadow Bans and crawler cache written Not touched
write Bans and crawler cache written Address list written (conditions only, other conditions kept)

A run is skipped, with an info line, when EDGE_BAN_MODE, EDGE_LOG_WORKSPACE_ID or EDGE_BAN_RULE_ID is empty. shadow and write are also skipped, with an error line, when the policy is empty or its Key Vault reference does not resolve. A rule write that gets 409 is retried by the next run. banmanual is never written.

The timer is registered only when EDGE_BAN_MODE is set. Deployed apps always set it.

Kill switch

Set app setting EDGE_BAN_MODE=off on the function app. The next hourly run writes the placeholder into banauto. No deploy is needed. Front Door takes 7 to 25 minutes to apply it.

App settings

Setting Bicep parameter Meaning
EDGE_BAN_MODE edgeBanMode off, shadow or write. Default off.
EDGE_BAN_SCHEDULE edgeBanSchedule NCRONTAB, six fields. Default hourly.
EDGE_BAN_POLICY edgeBanPolicySecretUri Key Vault reference to secret edge-ban-policy
EDGE_LOG_WORKSPACE_ID edgeLogWorkspaceId Customer id of eagle-logs-<env>
EDGE_BAN_RULE_ID edgeBanRuleId Resource id of the rulesdemidocs rule banauto

The test values are in azure/main.test.bicepparam.

Policy secret

Key Vault secret edge-ban-policy (demi-kv-<env>) holds JSON. Keys left out take the default.

Key Default Meaning
documentFloor 300 An address needs more distinct documents than this in 24 h to be judged
evidenceFloor 3 More browser evidence than this means leave alone
banHours [1, 4, 24] Ban length for the first, second and later offences
maxListLength 100 List cap
maxNewBansPerRun 20 Bans added per run
maxLookups 50 RDAP lookups per run
allow [] CIDRs never banned
crawlerAgents [] Extra user-agent words treated as declared crawlers

To set it, run from the devbox:

az keyvault secret set --vault-name demi-kv-<env> --name edge-ban-policy --file <policy.json>

Then list edge-ban-policy in optionalSecretNames of the environment's param file.

The edgeBans container

Cosmos NoSQL container edgeBans, partition key /id. One row per banned prefix: owner type, owner name, reason, offence times with document and evidence counts, banCount and expiresAt. The id is the prefix with / replaced by _. Every write resets a 7-day TTL. Row feed:crawlers holds the cached crawler ranges. The container is not application data and has no ACL.

Dry run

The CLI reads the access log and prints each candidate. It writes nothing: no Cosmos, no rule.

node src/scripts/edge-ban.js --dry-run --env test [--floor N]

--floor replaces documentFloor. The workspace comes from EDGE_LOG_WORKSPACE_ID, else from the workspace named eagle-logs-<env>. The caller needs Log Analytics Reader on it. Policy comes from EDGE_BAN_POLICY when it holds JSON, else the defaults.

Soak and promote

Each stage ends with an exit check. Each prod step needs explicit approval.

  1. Set EDGE_BAN_MODE=shadow and leave the list at the placeholder. Run 7 days, including weekday peaks. Exit: every row in edgeBans has zero browser evidence, is in no never-ban family, and either overlaps the scraper alert or has a reason a person reviewed.
  2. Set EDGE_BAN_MODE=write on test. The edge action edgeBanAction is already block, so every address the timer writes into banauto gets 302 to /blocked.html as soon as the rule updates. There is no count stage. Run 3 days. Exit: in the soak query (see Front-Door-Ban-Ladder), banauto matches only listed addresses, matches start within 25 minutes of a write, an expired ban stops matching, and block-page hits come only from listed addresses, with no support reports.
  3. Set EDGE_BAN_MODE=write on prod, with approval. Same exit check for 7 days.

Rollback at any stage: set EDGE_BAN_MODE=off. To stop turning requests away without waiting for a list change, set edgeBanAction=count and deploy.

⚠️ **GitHub.com Fallback** ⚠️