How CI works in noetl/ai-meta, what each check means, and what to do when it goes red. Also the fleet-wide record of which repos have a pull-request gate at all.

Source of truth is the repo, not this page: .github/workflows/ci.yml, .github/scripts/check-submodule-pointers.sh, .github/scripts/check-docs.sh.

ai-meta had zero workflow files. Not a weak gate — none. That is how #370 merged five submodule pointer bumps with nothing checking them.

A submodule pointer is a 40-hex string in a tree entry, and git validates nothing about it. It does not have to exist. It does not have to be pushed. It does not have to be on any branch. A typo, a stale local commit, or a bump to a never-pushed SHA all produce a repository that cannot be cloned — git submodule update --init fails for everyone, and nothing anywhere warns you at commit time.

Landed by #371 (merge a7d4b08e) and calibrated by #373 (merge 90f10022).

Workflow ci, two jobs, both on pull requests and on pushes to main.

The checkout uses fetch-depth: 0 (ancestry needs real history on both sides of the diff) and submodules: false — the pointer script fetches what it needs treelessly (--filter=blob:none), pulling the commit graph without file contents. Ancestry needs the graph and nothing else, so the whole 31-pointer sweep costs ~20 seconds.

Only pointers your change touches can fail the build. The full sweep is informational.

This split is deliberate. main already carries legacy drift (#372), and failing every unrelated PR on a pre-existing problem would park main permanently red — which is how a gate gets ignored, the exact failure the gate exists to prevent.

For each submodule path your change touches, in order. The first one that fails stops that pointer's remaining checks.

A submodule directory must stay mode 160000. If it becomes a regular file or a plain directory, someone deleted the submodule and committed its contents, or a bad merge flattened it.

::error::repos/server: submodule path changed to mode 100644 (expected 160000) — replaced by a file or directory

Deletion (000000) is allowed — removing a submodule is legitimate.

git ls-remote against the submodule's URL, before judging the pointer.

If the remote is unreachable the pointer is reported unverifiable, counted as neither pass nor fail:

⚠ remote not reachable from CI (private, renamed, or no credentials) — pointer UNVERIFIABLE here, not a failure

This check exists because the gate's first run got it wrong. It reported repos/noetl.io and references/chatui as "dangling, unpushed, or wrong SHA" when both repos were simply private — an unauthenticated runner cannot fetch any SHA from them. The gate asserted a cause it had never established.

Both submodules have since been removed (#377), so no declared pointer takes this branch today. It is kept for the next private submodule someone adds.

Coverage is now full: 31 of 31, zero unverifiable. It was 31 of 33 — the two private submodules (repos/noetl.io, references/chatui) could never be checked by an unauthenticated runner. Both were removed in #377 (merge 5bd14bdb), since neither carried a live dependency, and the hole closed with them. Measured on main by the runner itself, not locally:

submodules declared in .gitmodules: 31
RESULT: pass (examined=31 of 31 declared, 0 problem(s))
  check 1 fetchable+is-commit : 31/31 confirmed
  unverifiable (no CI access) : 0  <- NOT counted as pass or fail

So the token decision is moot — there is nothing left for a token to unlock. CHECK 0 stays anyway: adding one private submodule would silently reintroduce the false "dangling" verdict without it, and the unverifiable count going above zero is the signal that it happened. Either submodule can be re-added with git submodule add <url> <path> at the cost of reintroducing that pointer.

The pointer must name a commit reachable from some ref on the submodule's remote. This is the core bug class: a dangling, never-pushed, or mistyped SHA.

::error::repos/server: pointer 42e9276… is reachable from NO ref on [email protected]:noetl/server.git — dangling, unpushed, or wrong SHA

It first tries fetching the SHA directly; if the server refuses to serve a bare SHA (uploadpack.allowReachableSHA1InWant), it falls back to fetching refs and resolving the commit from them. Only reachable from no ref is a defect.

The pointer must be an ancestor of the submodule's main/master tip (already merged) or a descendant of it (ahead, not yet merged — allowed).

Neither means a detached bump onto a side branch:

::error::repos/docs: 04d2ebff… is NEITHER an ancestor NOR a descendant of main tip 61e742d3f9c2 — detached bump onto a side branch

This is the subtlest of the four and the one that found a real defect on main. ai-meta pointed repos/docs at a commit that exists only on the branch of open PR noetl/docs#188 — so a fresh clone resolved to content that is not shipped and that would cease to exist if that PR were rebased or squashed. The file it introduces is not on docs main at all. Tracked in #372.

Both main and master are tried, because the wiki submodules track master.

The old pointer must be an ancestor of the new one. If not, the bump regresses or rewrites the submodule — it moves backwards, or onto a rewritten history:

::error::repos/server: b512ed6cbd34 -> 492818ec532a is NOT a fast-forward — this REGRESSES or rewrites the submodule

If the old pointer is itself unfetchable, this is reported unprovable rather than failed — that is pre-existing, not something your change introduced.

One exception, added after the check deadlocked itself (#379). Moving a pointer off a side branch and back onto the tracked branch — exactly the fix CHECK 2 demands — is never a fast-forward, so CHECK 3 blocked it. CHECK 2 reported a defect CHECK 3 forbade correcting, which is where #372 sat. A non-fast-forward is now allowed when the new pointer is on the tracked branch and the old one was not, reported and counted separately:

✓ corrective re-point: 04d2ebffff04 was OFF main, 05d190b145db is ON it (not a fast-forward, allowed)
check 3 fast-forward : 0/1 confirmed, 0 unprovable (old pointer gone), 1 corrective re-point(s)

It keys on the old pointer being off-branch, which a genuine regression never is — a backwards move with both pointers on the branch still fails.

On changed markdown only, so it stays fast and cannot be reddened by the existing corpus:

• Balanced code fences. An odd number of ``` markers means an unclosed fence, which swallows the rest of the document when rendered.
• Relative links resolve. A link to a path that does not exist is reported. Bare page names with no slash or extension are skipped — those are legitimate inter-wiki links under this repo's conventions. External http(s) links are not checked, deliberately: network link-checking is the flaky kind.

Real output from a file carrying both defects:

  mode: markdown changed against 27330e2db734
  markdown files in scope: 1   <- the denominator
::error file=CI-RED-CONTROL.md::odd number of code-fence markers (1) — an unclosed fence swallows the rest of the document
::error file=CI-RED-CONTROL.md::broken relative link -> agents/rules/no-such-rule.md (resolved to ./agents/rules/no-such-rule.md)

  relative links checked: 1
RESULT: FAIL (files=1, links=1, 2 problem(s))

Errors are emitted as ::error file=…::, so they appear inline on the PR diff as well as in the job log.

Per representation-drift.md ("print the denominator"), each check publishes the population it measured, not only its findings:

RESULT: pass (examined=5 of 33 declared, 0 problem(s))
  check 1 fetchable+is-commit : 5/5 confirmed
  check 2 ancestry vs tip     : 5/5 confirmed, 0 skipped (no tip)
  check 3 fast-forward        : 5/5 confirmed, 0 unprovable (old pointer gone)
  unverifiable (no CI access) : 0  <- NOT counted as pass or fail
  mode                        : changed pointers, BLOCKING

A pass over nothing says so outright:

RESULT: pass (examined=0 of 33 declared)
  no check executed — this pass measured nothing

That line matters. A green result over zero pointers otherwise reads exactly like a green result over five — and a PR that changes no pointers legitimately produces the first one. If you want to know whether the gate actually checked your bump, read examined=, not the green tick.

An abort is a failure, never a silent pass: the script carries a COMPLETED marker and an EXIT trap, because its first version printed unbound-variable errors and still exited 0.

Run the same script locally — it is the same code CI runs, and it works on macOS (no mapfile, which is bash 4+):

# what a PR would check: changed pointers against the base
./.github/scripts/check-submodule-pointers.sh origin/main

# what the non-blocking sweep checks: all declared pointers (31 today)
./.github/scripts/check-submodule-pointers.sh

Then, by failure:

Per the standing constraint: gitlink-only, and never sweep sibling dirty repos/*. The shared checkout usually has many modified submodules, so committing with git add -A would drag unrelated pointers in. Use a fresh worktree off origin/main and set the single gitlink by object SHA:

git worktree add --detach /tmp/bump-wt origin/main
cd /tmp/bump-wt
git update-index --cacheinfo 160000,<new-sha>,repos/<name>
git diff --cached --raw          # MUST be exactly one :160000 160000 line
git commit -m "chore(sync): bump <name> to <short-sha>"
git push origin HEAD:main
cd - && git worktree remove --force /tmp/bump-wt

--cacheinfo needs the full 40-hex SHA; a 12-char abbreviation is rejected.

A check that has never been seen to fail is indistinguishable from one that cannot fail. Every control below ran as a real commit, and the red ones ran in GitHub Actions, not only locally.

CHECK 2 needs deliberate isolation: a side-branch bump also trips CHECK 3, so the control uses the side branch's merge-base as the old pointer. Then the fast-forward check passes and only ancestry can fail. Without that, CHECK 2 stays unproven while appearing covered.

noetl/noetl carried 214 open Dependabot alerts. The arc down, and where it stopped, is a useful worked example of the difference between refreshing a lock and changing what the code depends on.

The 5.x CLI bump cleared zero. It was worth landing — the wheel is tagged 5.0.0 and was built from a 4.21.0 CLI, so the artifact and its engine disagreed — but rpassword, rustls-webpki, jsonwebtoken, lru and async-nats are at byte-identical versions before and after, even though noetl-tools crossed a major version (3.26 → 4.0.3). A major bump of an intermediate crate is not evidence that its own dependencies moved.

async-nats is not a dead transport. The claim that four advisories came from "a transport deleted at T5" conflated two different things:

• NATS as the platform's internal bus — deleted at T5. Gone.
• kind: nats as a tool, plus NATS as a subscription source — live, shipped, and always compiled in (noetl-tools Cargo.toml: "NATS is always compiled in (async-nats is a core dependency)"; src/tools/nats.rs, src/tools/source/nats.rs, registry.rs registering "nats", and the spool/ circuit-breaker paths).

The noetl/cli file the removal was aimed at mentions async_nats in two doc comments and nothing else — deleting them clears nothing. The real edge is noetl-tools → async-nats 0.38 → rustls-webpki 0.102.8, and the fix is a version bump, not a removal: async-nats ≥ 0.47 requires rustls-webpki ^0.103.10, which satisfies all four advisories while keeping the tool kind. That is nine minor versions of a 0.x crate across four modules plus a pinned-time workaround, so it stays tracked rather than forced.

The general lesson, which generalises past this repo: "this dependency is unused" and "this dependency's feature was retired" are different claims, and a doc comment is not a dependency. Check the Cargo.toml and the registry before proposing a removal — #385.

None were fixable in the wrapper: it has exactly two direct dependencies (noetl, pyo3), so every one was transitive and every patch a breaking bump in a repo one or two levels up. All seven are now cleared (#385).

Measured before changing anything, in every case:

• async-nats 0.38 → 0.47 — one break, at src/tools/nats.rs:639: ObjectMetadata gained fields, so the struct literal needed chunk_size: None, metadata: Default::default(), headers: None. It also required rustls-webpki ^0.103.10, which is the whole point. A time pin came out with the reason recorded.
• jsonwebtoken 9 → 10 — ⚠ broke at runtime, not compile time. Version 10 requires an explicit CryptoProvider; the code compiled clean and failed when exercised. One existing test caught it. Fixed with features = ["rust_crypto"]. A green cargo build would have shipped it.
• rpassword 3 → 7 — clean drop-in across four major versions; the read_password() signature in use was unchanged.
• ratatui 0.26 → 0.30 — 2 errors at one line plus one deprecation. Backend gained an associated Error type, so run_app<B: Backend> needs where <B as Backend>::Error: Send + Sync + 'static; Frame::size() → Frame::area(). It also exposed a duplicated crossterm (ratatui on 0.29, cli declaring 0.27 directly) — two independent copies of raw-mode state on the same tty. Collapsed to one.

ratatui 0.30 does not drop lru — the new ratatui-core 0.1.2 still depends on it. The advisory clears on version (0.18.5 is above the vulnerable < 0.16.3), not on removal. Anyone checking "did lru go away?" will find it present and should not treat that as a failed fix.

"All 7 cleared" was verified by parsing both locks with tomllib and comparing each alert's first_patched_version against the resolved version — and by running the identical logic against the previous lock, which reports 4 cleared / 3 residual. That control is load-bearing: a parser that silently loses packages treats "not found" as "cleared" and produces a clean sweep for the wrong reason, which is exactly how an earlier count in this arc came out wrong. Parsed package counts (456 → 524) are published alongside the result, because "7 cleared" is also consistent with a scan that examined seven things.

#395, closed 2026-10-04. Clearing the lock took the alert count to 0; it did not change what anyone installs. The published wheel stayed noetl 5.0.0 from 2026-07-26 — with the 4.21 CLI and every vulnerable crate — because pypi.yml publishes only on a pypi-v* tag and none had been cut.

Published: noetl 5.1.0 · https://pypi.org/project/noetl/5.1.0/

twine upload --non-interactive --skip-existing dist/*

The published version comes from packaging/pypi/pyproject.toml and packaging/pypi/Cargo.toml — not from the tag, which only triggers. Nothing bumps those automatically: semantic-release's prepareCmd rewrites the root pyproject.toml and packaging/pypi/** is outside its assets list entirely.

So tagging without bumping them builds the already-published version, --skip-existing skips every artifact, and the job reports green. A release that publishes nothing and says it succeeded — || true in PyPI form. Nothing cross-checks the tag against the built version, and the verify step only prints noetl.__version__.

Tracked as #400. The operational rule: a green publish job is not evidence that anything shipped — read the upload/skip counts. For 5.1.0: 4 × Uploading noetl-5.1.0-*, Skipping 0, already exist 0, and PyPI went from "5.1.0 absent" (baseline captured before tagging) to 4 files.

⚠ An earlier note here (and #395's own body) said pyproject sat below the published 5.0.0 and the version line was broken. That compared the wrong file. They are independent package definitions, and the wheel's version was always the packaging/pypi/ one.

5.1.0 is minor: above 5.0.0; not a patch because the embedded CLI moved 4.21.0 → 5.0.1, which changes what the noetl command does; not major because the package's own API is unchanged. Deliberately not aligned to the CLI version — packaging/pypi/README.md documents that the two are separate, and they already were.

Three independent means, on files downloaded from PyPI with sha256 matched to PyPI's own digests:

1. published sdist Cargo.lock — 524 packages parsed, all four patched, none in a vulnerable range (control: the same logic flags rustls-webpki 0.102.8)
2. published wheel binary — strings shows rustls-webpki-0.103.13, rpassword-7.5.4, lru-0.18.5
3. publish-run compile log — 3 compile lines each, one per platform, including the jsonwebtoken 10.4.0 that left no string in the stripped binary

⚠ pypi.yml builds without --locked, so a wheel's resolution is not guaranteed to equal the committed lock. That is not theoretical: the 5.0.0 run compiled both rustls-webpki 0.102.8 and 0.103.13. Which is also how a strings probe briefly convinced me the old artifact was fine — finding a patched copy is not evidence that a vulnerable one is missing.

#395 closed the seven Dependabot alerts covering four crates. A full RustSec scan of the same shipped lock found 5 patchable vulnerabilities in 4 other crates — event-listener (unsoundness), h2 (DoS), rustls (crypto-failure), quick-xml ×2 (DoS) — plus rsa (Marvin, no patch) and informational unmaintained notices.

Dependabot reports 0 for that lock. Two oracles, same input, different answers (#402).

So "0 open Dependabot alerts" is not "no known Rust vulnerabilities". The #395 fix was verified against the oracle that raised the alarm, and a second oracle sees more — the same shape as the issue itself, one level out.

h2 0.4.15→0.4.19 and rustls 0.23.42→0.23.45 clear their advisories; event-listener 5.4.2 and rustls-webpki 0.103.15 came with the re-lock (noetl#706, bumped as #404). On the same lock, right now: Dependabot 290 alerts all fixed, 0 open; RustSec 3.

⚠ A 3 is only a result if the scanner was shown able to find 5. Reporting a low number from an instrument that has never produced a positive on this input is the same shape as every inert gate on this page, so the previous shipped lock was scanned first as a positive control and had to report exactly the five:

5.1.0 shipped lock   5   h2, quick-xml x2, rsa, rustls      <- positive control
5.1.1 shipped lock   3   quick-xml x2, rsa
denominator          1270 advisories parsed (0 unparsed), 574 packages in the lock

That denominator is load-bearing twice over. The advisory files are .md with TOML front matter, not .toml, so a glob for crates/*.toml matches nothing and prints a clean repo — the parsed > 1000 assertion is what refused it. And a grep -m1 patched in my own display of rustls-webpki's advisories showed only >= 0.104.0-alpha.6, making the scanner look like it was missing three hits; the real field is [">= 0.103.12, < 0.104.0-alpha.1", ">= 0.104.0-alpha.6"], which 0.103.15 satisfies. The truncation was in the instrument built to check the instrument.

The remaining 3 are accepted waits, not blockers — #402 closed, and they are tracked at #406. Their causes differ, which is the part worth not conflating:

• quick-xml ×2 — not waiting on upstream. The patch is >= 0.41.0, reachable only via object_store 0.14, which pulls aws-lc-rs in beside ring and leaves rustls with two registered CryptoProviders and no default; five TLS tests fail at runtime, not compile time. The missing piece is our decision about which crypto implementation ships in the wheel.
• rsa/Marvin — genuinely upstream. patched = []; there is no fixed release.

So 3 is the expected reading on a current published lock. A release here is not clean at 0; it is clean at 3 with those three named. Do not hold a release on them.

One thing that did go right by construction: the shipped sdist's Cargo.lock was byte-identical to packaging/pypi/Cargo.lock on the released commit. #395 is why that is checked at all.

Audited 2026-10-02. The question asked was "does anything trigger on pull_request?" — not "are there workflow files". Several repos had workflows that run only on a tag or a push to main, which gates nothing before a merge. 16 repos examined.

clippy could not fail on five Rust repos. server, worker, cli, tools and gateway all ran cargo clippy … || true — the step appeared in every PR log, passed any "do we run clippy?" grep, cost its full runtime, and could not ever fail. Reachability-vs-existence exactly. #374.

All five now gate (closed 2026-10-03). Measuring first explained the || true: a bare -D warnings gives 22 errors / 14 lint kinds on server and 15 / 7 on worker, so an outright flip would have landed red on arrival. Both now gate with the existing kinds allow-listed by name — a new lint of any other kind fails — and the debt sweep is #378. Proven by planting a non-allow-listed clippy::len_zero: the clippy step went red while cargo test stayed green, isolating the lint gate.

tools, gateway and cli were then measured and flipped in turn — see Paying down the inert clippy gates below for what each cost and the four measurement errors the work surfaced.

noetl/noetl needed no test suite — it needed less Python. The audit listed it as "no pytest anywhere" as though that were a gap. The intended state is a thin wrapper around the Rust noetl CLI (packaging/pypi/, maturin/PyO3, published as the PyPI noetl package), so there is almost nothing left to test. #382 removed the residue — 77 Python files, 90 → 9, −16,961 lines — keeping only the wrapper, the live schema_ddl.sql that repos/ops provisions from by path, and the test that guards that DDL. Its gate is scoped accordingly: the DDL contract, plus release prerequisites (uv version exits 0; the wrapper still declares noetl/ntl; and noetl/ contains no Python).

Final state (#701, head e82697af): Python 90 → 3 files, +633 −19,095 across 104 files. The three survivors are the wrapper, a package marker, and the DDL contract test. The lock went 241 → 36 packages, which addressed all 183 of the repo's pip Dependabot alerts (164 whose packages are gone outright, 19 fixed by version); the residual 31 are npm and Rust lockfiles a uv refresh cannot reach. Dropping the IBKR scripts broke a repos/ops playbook that invoked one by absolute local path — handled in noetl/ops#318 with a loud, documented failure rather than a silent break, disposition in #383.

⚠ That last guard set exists because the removal itself broke the release pipeline: dropping the setuptools package config made uv version exit 1 while still rewriting the version, so the output looked right and only the exit code showed it. The next release would have failed at prepare. A guard written from a mistake you actually made is the kind most likely to fire again.

Wiki repos cannot run GitHub Actions at all. *.wiki.git has no runner, so "add CI to the wikis" is not achievable as stated — that was the finding, not a shortfall. ✅ Resolved 2026-10-03 by .github/workflows/wiki-docs.yml: a scheduled ai-meta job that clones each wiki and runs check-docs.sh over it. Same rules, one day late rather than pre-merge. See The wiki sweep below.

#232 is stale-corrected. It recorded that no cargo test ran in CI on any Rust repo. That is no longer true: all seven run cargo test --all-targets on pull_request. --all-targets matters — plain cargo test skips targets it compiles, which is how a test that does not even compile can sit on main looking fine.

#374 closed 2026-10-03. Each repo was measured before being flipped, because flipping without measuring is what produced || true in the first place.

cli's two exceptions are large_enum_variant (boxing Tool's variants would change the published DSL schema, since it is also derived under the json-schema feature — a downstream contract change, not a cleanup) and result_large_err (the large Err is tonic::Status, upstream, at a test-helper site).

too-many-arguments-threshold = 8 in a new clippy.toml covers cli's two 8-arg functions. That is deliberately a threshold rather than an -A: a function growing to 9 arguments still fails, verified with a 9-arg probe that fired (9/8).

An unexplained -A is how a gate quietly stops gating — it reads as configuration rather than as debt, and nothing ever revisits it. Every exception in these workflows states what it is and why it cannot simply be fixed, so the next reader can tell a real constraint from an abandoned one.

Worth recording, because every one produced a confident number that was wrong:

1. A -D warnings run is not an inventory. On cli it reports 15 errors; the real count is 39. -D warnings promotes warnings to errors, so compilation aborts at the first failing crate — noetl-executor failed first and the main noetl crate, holding 23 of the lints, was never linted at all. Sizing the work off the gating run understates it by more than half.
2. clippy serves cached warnings. A "39 → 6" reading kept printing (8/7) after clippy.toml raised the ceiling to 8. The control run on a recompiled crate printed (9/8), which is the only reason it was caught. The true figure after touching every .rs file is 39 → 4.
3. clippy --fix introduced a hazard worse than the lint it removed. Its manual_flatten fix turned if let Ok(line) into reader.lines().flatten(), which is lines_filter_map_ok: flatten() loops forever if the reader repeatedly returns Err. An auto-fix is a suggestion, not a correctness proof. Both sites are now map_while(Result::ok).
4. A local RED proof returned exit=127 three times and looked like a pass. The gate command was held in a shell variable and run as $CMD; zsh does not word-split, so the whole string became one command name and clippy never ran. Clean, planted and restored all looked identical — gating on "0 errors in the log" instead of the exit code would have recorded a clean proof from three runs that invoked nothing.
5. git checkout -- <file> to remove a probe reverted four real fixes in the same file. The assertion only checked that the probe was gone, so it passed.

A gating clippy is only as stable as its lint set, and the lint set comes from the toolchain:

Without a pin, a Rust release can add a lint and turn main red with no code change. Not hypothetical: the tools RED-proof run logged stable-x86_64-unknown-linux-gnu updated - rustc 1.99.0 (from rustc 1.98.1) during the run.

It also means a local green is not evidence about CI — the local toolchain here was 1.97.1 while CI ran 1.99.0, two minor versions apart and a different lint set. The CI run is the measurement. Pin decision tracked on #378.

cli's first gate PR reported a red test workflow that looked like a clippy failure and was not. ci/check-member-versions.sh (#331) refused the head because the PR changed executor/ while leaving executor/Cargo.toml at 0.10.0 — the release would have cut a tag and published a noetl-executor without the change.

It runs before the toolchain install, so build, test and clippy were all reported skipped. Two lessons: the guard works, and a pre-toolchain gate makes every later step's result unavailable rather than green. The companion RED-proof branch failed at the same step and therefore proved nothing about clippy until it was rebuilt on the fixed head.

Closed 2026-10-03 under #375. Each was measured green before being made gating, and each was proven able to fail.

All 36 "failures" are Helm templates — Go template syntax is not valid YAML by design. Shipping that would be a gate crying wolf on arrival, which is how a check gets switched off. They are covered by helm lint instead, and a control proves the exclusion is not a blind spot: breaking a template fails charts while yaml stays clean.

Templates are identified by a sibling Chart.yaml, never by a path pattern, so adding or moving a chart cannot silently drop it from coverage. ops already has that scar: ops#252's PodMonitoring enumerated four worker app names, two of which never existed, while omitting one that did — and partial coverage emits no error at all.

docs/playbooks/** is generated from fixtures/playbooks, and npm run build regenerates it first — so the published site was always correct while the committed copies rotted. They had: the committed index claimed 144 playbooks against 162, with seven whole category pages never committed at all. Invisible to anyone reading the site; plain to anyone reading the repo.

The check is git status --porcelain docs/, not git diff, precisely because most of that drift was files that had never been added — which a plain git diff does not see.

Also on e2e: onBrokenLinks was 'warn', so a build could not fail on a broken link. In a docs repo, the thing most likely to break could not turn the gate red. Measured at 0, then flipped to 'throw'.

It shipped with two tiers. FAIL covered unambiguous corruption — a .deb whose SHA256 no longer matches its index entry is a broken install for everyone. REPORT restated three real defects on every run, because the publishing pipeline could not satisfy them yet:

• 22 of 23 .debs were in no index — dpkg-scanpackages ran without --multiversion, so apt-get install noetl=<older> could not resolve.
• arm64 was in pool/ but indexed nowhere — binary-amd64 was hardcoded.
• Release carried none of its 8 identifying fields, and listed itself at 38 bytes against a real 1792 — apt-ftparchive release D > D/Release scans the directory it writes into.

⚠⚠ The finding behind the finding: ops/scripts/publish_apt.sh already did all of this correctly — and nothing invoked it. The release pipeline used its own inline commands. A correct implementation sitting beside the less correct one that is the only one that runs: "exists but is not reached", one level up from code.

They were reported rather than failed because the arm64 disposition was a human call — the newest arm64 .deb was 2.8.7 against amd64's 5.0.1, so indexing it would have made arm64 look supported at a version years behind. Reported rather than dropped because a known defect with no recurring output becomes an unknown defect.

cli#93 took the third option: build a current arm64 .deb every release. ubuntu-24.04-arm runners were already available, so build-deb became a two-entry matrix rather than a cross-compile, the apt job discovers architectures from pool/ and runs dpkg-scanpackages --multiversion --arch once per architecture, and apt-ftparchive writes Release to a temp path and mvs it in. Measured on the live published repo after v5.0.3:

.deb in no index        22 of 23  ->  0 of 26
architectures indexed   [amd64]   ->  [amd64, arm64]
Release header fields    1 (Date) ->  8
Release self-entry      present   ->  absent, both suites
newest arm64             2.8.7    ->  5.0.3

Nothing was deleted — the old arm64 files stay installable as the genuinely older releases they are, which is what --multiversion is for.

⚠⚠ And the one that was never reported: without --arch, dpkg-scanpackages writes every .deb into whichever index it is told to produce — architecture is ignored. Latent while only amd64 shipped, and it would have started serving arm64 binaries to amd64 clients the moment arm64 existed. apt#3 adds that as a fourth check and promotes all three originals to FAIL, each injected one at a time against the real published tree to prove it fires:

drop all but one stanza from an index   -> exit 1
put an arm64 stanza in the amd64 index  -> exit 1
remove Codename from Release            -> exit 1
re-introduce the Release self-entry     -> exit 1
all restored                            -> exit 0

A promoted check that cannot fail is just a comment, which is what the REPORT tier already was. One deliberate omission: Version is not in the required header set — it is optional in Debian and the generator does not emit it, while Suite and Codename are what apt matches on. Requiring a field the correct output lacks is how a gate earns a permanent || true. #390 closed.

Each check fails below a floor as well as on findings, because 0 failures is also what a check that examined nothing looks like. Proven by running each against an empty repo: all floors fire. apt is the sharpest case — a check over an empty pool otherwise reports exactly the same clean result as a healthy repository.

The e2e gate's first CI run failed on my own workflow: ModuleNotFoundError: No module named 'yaml'. The docs job set up Python without installing PyYAML while the files job did; locally yaml was already importable. Third instance in this arc of a local pass that said nothing about CI.

.github/workflows/wiki-docs.yml + .github/scripts/check-wikis.sh, daily at 06:40 UTC, plus workflow_dispatch, plus a pull_request trigger on its own files so the sweep cannot be broken silently and discovered a day later by a cron run nobody watches.

wikis declared:  15
wikis checked:   15
not cloneable:   0
markdown files:  243
relative links:  1684
RESULT: pass

It started at 3 findings, not 0. Two genuinely broken links that nothing had ever checked, since the wikis have no runner:

• It checks the live wiki, not the submodule pointer. The point is to notice a wiki broken right now; a pointer can be months behind.
• Anonymous HTTPS clone. The declared submodule URLs are SSH and a runner has no key. These are public wikis, so no token is involved.
• The wiki list is derived from .gitmodules, never enumerated.
• It fails below a floor — under 10 wikis checked, or 0 markdown files examined.
• Bare page names stay exempt, because wiki inter-page links are slugs rather than paths; flagging them would make the sweep red on arrival.

The sharpest instance of this whole arc's theme. trap 'cleanup; on_exit' EXIT replaced the earlier trap, so rc=$? inside on_exit captured the status of cleanup — always 0 — rather than the script's. Two controls printed RESULT: FAIL while exiting 0: a green job announcing its own failure, because CI reads the exit code and not the text.

It was caught only because the controls asserted the exit code rather than grepping the output. Had they grepped for "FAIL" they would have passed, and a permanently-green sweep would have shipped looking like thorough work.

The general form: when testing a gate, assert the thing CI actually consumes.

#378, 2026-10-03. Fleet -A entries 23 → 2.

⚠ Correction to an earlier figure on this page's sibling issue: the count was reported as 27. It was 23 — the grep that produced 27 matched -A below, -A entries and -A is from the surrounding comment prose rather than the command. Parse the run: block, not the file.

The best single argument for this work. The entry was suppressing the dead_code report on labels_are_a_closed_set and insert_column_and_bind_counts_agree — #361, now closed.

The mechanism in both files: a stray #[test] sat above a doc block with a second #[test] below it, so both attributes bound to the earlier function — reported as duplicate_macro_attributes, which was also allow-listed — while the next function was left unmarked and inert. The two entries that hid it sat adjacent in the list.

A second gap came out with it: unused_variables covered high, a member of a deliberate up(500 ns)/down(499)/nearly(999)/high(798) family in which every other member is asserted. The lint's own suggestion — rename to _high — would have preserved the gap and marked it intentional.

The general shape: an allow-list does not just defer debt, it hides the next finding of the same kind.

Three classes survived, each scoped to its site with its reason:

• dead_code on two worker statics — not dead. Each is read only from a #[cfg(not(test))] arm, and clippy --all-targets compiles the test cfg, so the reader is compiled out and the static reads dead there while being live in every shipped build. Verified by cargo clippy --lib (non-test cfg) reporting neither.
• non_snake_case on three server test names — the capital is the word that makes the name state its own verdict (..._BOTH_gates_are_on, ..._is_REFUSED_not_recovered, ..._is_NOT_wired_to_...). Snake-casing makes each read as the opposite claim.
• cli's large_enum_variant / result_large_err — the documented reason was "boxing would change the published DSL schema". That was then tested: boxing ToolSpecDoc::One compiles cleanly and makes tests/schema_guard.rs::committed_schema_is_not_stale fail. The schema does change, despite Box<T> being transparent to serde.

The test for a justified allow is mechanical: remove it and see whether the gate fails. Every surviving allow was removed individually and each produced exit 101; all restored, exit 0. An allow that suppresses nothing is indistinguishable from debt already paid, and should be deleted rather than inherited.

In an issue about inert gates:

1. #[allow(clippy::assertions_on_constants)] placed directly on an assert! invocation is silently ignored — rustc: "the built-in attribute allow will be ignored, since it's applied to the macro invocation assert". The lint still fired while a comment claimed a scoped suppression that did not exist. Attributes belong on items.
2. The first load-bearing battery reported 2 of 4 allows as decorative. Wrong: the shell split cases on :, which also appears inside #[allow(clippy::...)], so the wrong text was removed and the label printed empty. Re-run with a safe delimiter and an explicit before/after count assertion, all four were load-bearing.

Five assertions over constants moved from test-run time to build time. ⚠ They fail cargo build, not cargo check — check is metadata-only and skips const evaluation of function-body inline consts, which is why a first mutation test of this read as a pass. CI runs build --all-targets, so they gate.

Three worker sites could not take the const form at all: their message interpolates the constants, and a const-block assert! cannot format (E0015). Tried and rejected by the compiler, not assumed.

server, worker and tools had no rust-toolchain.toml while CI installs dtolnay/rust-toolchain@stable — so a Rust release could redden main with no code change, which matters most once a gate has no allow-list to absorb a new lint. All three now pin 1.99.0, the version CI had already resolved to.

⚠ cargo honours the file over the rustup default — verified empirically, because the obvious signal is misleading: cli (pinned 1.91.1) emitted a CI lint citing rust-clippy/rust-1.91.0, while unpinned tools emitted rust-1.99.0 from the same action on the same day. The action's console banner prints the version it installed, which is not what cargo used. Reading that banner is what briefly convinced me pinning did not work at all, and that claim had to be retracted. The lint's own URL is authoritative, because it comes from the clippy that ran.

ehdb and signal-mesh were the last two unpinned repos, and both gate with -D warnings and no allow-list — so neither had anything to absorb a new lint. Pinning them turned up a live case:

• rustc 1.99.0 deprecated Atomic::fetch_update (renamed to try_update).
• crates/ehdb-l0/src/fault.rs:68 calls fetch_update.
• ehdb's last green CI run was 2026-10-01, and that run installed rustc 1.98.1 — the deprecation did not exist yet.
• The action rolled to 1.99.0 on 2026-10-03, verified across five other repos' runs the same day.

So ehdb's next push would have failed the gate, with no commit at fault and nothing regressed. It was found by pinning, not by the build — the failure was simply waiting.

The fix was not try_update. That method does not exist before 1.99, and the workspace declares rust-version = "1.82"; adopting it would raise a declared MSRV by 17 releases to silence a pure rename with identical semantics. A scoped #[allow(deprecated)] carries that reason instead, placed on the expression rather than the crate so a different deprecated call still fails. Proven load-bearing: removing it reproduces the exact error.

⚠ One more thing worth checking when pinning a repo that gates cargo fmt --all --check: the pin also changes rustfmt, and a different rustfmt reformats. 1.99.0 ships rustfmt 1.10.0; both ehdb and signal-mesh pass --check under it, verified before the pins landed. A pin that fixes the lint floor and breaks the fmt gate is not an improvement.

A gate that is red the day it lands is a gate people learn to skip. Every check added above was measured against the target repo's main first:

Copying ehdb's ci.yml into doctor unchanged would have produced a job that was red on arrival, purely from formatting.

⚠ Do not clear a Rust fmt backlog with cargo fmt --all on these crates: rustfmt 1.9.0 reformats far beyond the intended file — one attempt rewrote 39 files and broke a test. Format single paths with rustfmt <path>.

• Issue Tracking — pointer bumps reconcile code, wiki and issue in one change set
• Deployment Validation — kind validation before prod
• Wiki Convention — which wiki a page belongs on
• agents/rules/representation-drift.md — print the denominator; reachability vs existence