AAP Runbook - SS67/project-docs GitHub Wiki

1. AAP Operations Runbook

Companion to `RedHat_Ansible_Automation_Platform_setup_reference`. That doc carries the build: topology, VIPs, ports, file paths, F5 config. This doc carries operations: how to check health, what breaks, how to fix it, what is still open.

Read both. This one assumes the other is present and does not repeat it.

Derived from the Sep 2 2026 execution node incident. Everything here was verified on the running cluster, not taken from documentation.

1. How to use this doc

If you are an agent with this in project files, read it end to end before acting on the platform. It carries current state, known failure modes with real symptoms, and constraints that will cause damage if ignored.

If anything below contradicts what you observe on the live cluster, the cluster is right and this doc is stale. Flag it and ask before proceeding.

1. Constraints for automated access

Applies to any MCP, agent, or script with access to this platform.

Start read only. Health queries, log reads, `awx-manage list_instances`, status checks. Add write paths one at a time, deliberately.

Never write to Postgres directly. The DB sits behind Patroni. Only the leader accepts writes. Writing to a replica or bypassing the VIP corrupts cluster state. If a task seems to need direct DB access, it is the wrong task.

Never restart Patroni or etcd from automation. Manual, with a human watching, or not at all.

Never disable an instance without checking for running jobs first. Disabling sets capacity to 0. Work in flight fails.

Never change forks or capacity on a running job template. Change it, then run. Not during.

One node at a time. Two exec nodes exist. Taking both out at once leaves no capacity. Same for controllers.

Confirm before any `systemctl` on gateway or DB nodes. Those carry the F5 pool. An unannounced restart drops a pool member.

1. Current state

As of Sep 3 2026.

1. Nodes

The awx UID differs per node. 1000 on proden01, 1002 on proden02. Controllers unverified. Never hardcode the UID in a script or unit file. Always resolve it:

UID_AWX=$(id -u awx)

1. Instance groups

`fresh_sync_cp01` exists because the Freshservice tools depend on local disk at `/var/lib/awx/fs_snapshot` and `/var/lib/awx/fs_reports`. Do not repoint those templates without migrating that state.

Controllers remain in `default`. Any template not explicitly pinned can land on a controller. That is a deliberate open decision, see Open Work.

1. Filesystems, exec nodes

`/var/lib/containers`, `/var/lib/awx`, and `/var/lib/receptor` all sit on `/var`. Not separately partitioned. VG has roughly 30 GB free after the extend.

1. API path

AAP 2.6 namespaces the controller API behind the gateway.

https://aap.example.com/api/controller/v2/...     correct
https://aap.example.com/api/v2/...                404 at envoy
https://aap.example.com/api/gateway/v1/ping/      platform health

1. Health checks

1. Platform

curl -ks https://aap.example.com/api/gateway/v1/ping/ | jq

Expect `status: good`, `db_connected: true`, `proxy_connected: true`.

1. Instances

sudo awx-manage list_instances

Run from a controller. Check for DISABLED tags, capacity 0, and stale heartbeats.

Capacity is a scheduling number, not a fork budget. It is derived from memory by default, roughly (MB minus 2048) divided by 100. On 32 GB that reads about 297 while CPU capacity at 8 cores is 32. Reading capacity as a fork limit is what caused the Sep 2 incident.

1. Execution node, full check

Run on the node. Resolve the UID first.

UID_AWX=$(id -u awx)

systemctl status user@${UID_AWX}.service
ls -ld /run/user/${UID_AWX}
systemctl status receptor
su - awx -c 'podman info | head -20'
su - awx -c 'podman images'
su - awx -c 'podman ps -a'
free -g
df -h /var /tmp /var/log
cat /proc/pressure/memory

All must pass. Receptor being active proves nothing on its own, see Failure Mode 1.

Use `su – awx`, not `sudo -u awx`. The latter skips PAM, so no session and no runtime dir is created. It fails on a healthy node and tells you nothing.

1. Memory, live

free -g

Read the available column, not free. Free drops to 1 to 2 GB routinely while available stays healthy, because buff/cache is reclaimable. Acting on the free column leads to false alarms.

Better signal:

cat /proc/pressure/memory

If `some avg60` climbs past 10, the kernel is stalling processes on reclaim. Near zero means no pressure regardless of what free shows.

1. Real memory footprint

RSS double counts. Ansible forks share pages with the parent through copy on write, and RSS charges shared pages in full to every process. Summing the RSS column overstates usage by several times.

Use cgroup accounting, which counts each physical page once:

UID_AWX=$(id -u awx)
cat /sys/fs/cgroup/user.slice/user-${UID_AWX}.slice/memory.current
cat /sys/fs/cgroup/user.slice/user-${UID_AWX}.slice/memory.peak

Or sum Pss, which splits shared pages proportionally:

for p in $(pgrep -f ansible-playbook); do
  awk -v p=$p '/^Pss:/{s+=$2} END{print p, s/1024" MB"}' /proc/$p/smaps_rollup
done

Peak divided by fork count gives real incremental cost per fork. That is the number to size from.

1. Failure modes

1. 1. User manager death, silent node failure

This is the one that cost 30 hours. Highest priority to detect.

What happens. A system wide OOM kills `systemd —user` for the awx UID. `[email protected]` is static with no `Restart` directive, so it does not come back. Logind tears down `/run/user/UID`.

Receptor is a separate system service managed by PID 1. It survives, keeps heartbeating, keeps accepting work units. Rootless podman needs `XDG_RUNTIME_DIR` and cannot start containers without it.

Result: the node reports healthy to the controller and fails every job it accepts.

Observed Sep 2. proden01 OOMed at 02:49:59. AAP kept dispatching for 13 more minutes, nine job attempts, all failed. Node then sat in the pool doing nothing for 30 hours. proden02 OOMed at 09:54:15 with the same outcome.

Symptoms.

systemctl status user@${UID_AWX}.service   -> failed or inactive
ls -ld /run/user/${UID_AWX}                -> No such file or directory
systemctl status receptor                  -> active (running), looks fine
awx-manage list_instances                  -> heartbeat current, capacity normal

Podman fallback debris appears in `/tmp`:

/tmp/storage-run-<UID>

That path is created when podman loses its runtime dir and tries to continue. Its presence is a strong signal.

Fix.

UID_AWX=$(id -u awx)
sudo systemctl start user@${UID_AWX}.service
ls -ld /run/user/${UID_AWX}
sudo systemctl restart receptor
su - awx -c 'podman info | head -20'

Then clean orphaned job dirs. Nothing running:

du -sh /tmp/awx_* /tmp/ansible_runner_registry_* /tmp/storage-run-${UID_AWX}
rm -rf /tmp/awx_* /tmp/ansible_runner_registry_* /tmp/storage-run-${UID_AWX}

Never wildcard those paths while jobs are running. Leave `/tmp/receptor` alone, it is live state.

Detection. No native AAP check catches this. Add to monitoring, per exec node:

UID_AWX=$(id -u awx)
systemctl is-active user@${UID_AWX}.service
test -d /run/user/${UID_AWX}

Both must pass. Alert on either failing.

Why linger matters. `/var/lib/systemd/linger/awx` tells logind to start the user manager at boot with no login session. Without it, a service account never gets one. Verify it exists on every exec node.

1. 2. Boot ordering race

Observed on proden01, Sep 3.

receptor              active since 08:06:42
[email protected]     active since 08:06:43

Receptor comes up one second before the user manager. There is a window on every boot where the node accepts work and podman cannot run.

Confirmed behaviour: after the resize reboot, jobs kept landing on controllers until receptor was restarted manually.

Workaround. After any exec node reboot, confirm the session then restart receptor:

UID_AWX=$(id -u awx)
systemctl status user@${UID_AWX}.service
ls -ld /run/user/${UID_AWX}
sudo systemctl restart receptor

Not yet fixed. A drop in with `[email protected]` and `After=` would enforce ordering, but it is not Red Hat recommended, has no KCS backing, hardcodes a UID that differs per node, and may be overwritten on upgrade. Open with Red Hat before applying.

1. 3. Execution node OOM

Root cause of the Sep 2 incident.

Config that caused it: 1000 hosts, forks 100, job slicing 1, full fact gathering, on 4 cores and 15 GB.

Slicing 1 means one indivisible job on one node. No distribution. The second exec node sat idle.

Kernel evidence:

constraint=CONSTRAINT_NONE, global_oom

System wide, not a cgroup limit. Systemd logged `Consumed 7.4G memory peak` on the user slice.

Current mitigations. Forks 25, slicing 4, trimmed `gather_subset`, nodes at 8 cores and 32 GB, `patching` instance group.

Diagnosis.

journalctl -k --since "<window>" | grep -iE "invoked oom-killer|Out of memory|Memory cgroup"

`Memory cgroup out of memory` means a cgroup limit was hit. Absent that line, it is system wide and the node is undersized or the job is oversized.

Check what got killed. If the kill list includes `systemd`, `(sd-pam)`, or `dbus-broker` under the awx UID, Failure Mode 1 has also occurred and the node is now silently broken.

1. 4. Control plane fallback

Controllers are `node_type=hybrid` and members of `default`. When both exec nodes hit capacity 0, jobs targeting `default` land on a controller.

Observed Sep 2. After the exec nodes were disabled, the patch job ran on prodac01 at forks 100. prodac01 is a gateway pool member, a controller, and the sole member of `fresh_sync_cp01`. An OOM there costs a controller, a gateway path, and the entire Freshservice pipeline.

Current mitigation. Patch templates pinned to `patching`. Controllers remain in `default` for everything else.

Detection. Check the Execution Node field on job details. If production work is running on prodac01 or prodac02, ask why.

1. 5. No job failover between execution nodes

Not a bug. AAP does not migrate jobs between execution nodes. A job is assigned at dispatch and stays there. Node dies, job dies.

Running two exec nodes gives distribution of new work, not failover of running work.

Job slicing is the only mechanism that limits blast radius. Slicing 4 means one node failure costs one slice, not the whole run.

1. 6. Disk exhaustion

Each job private data dir under `/tmp` carries a full project copy. Concurrent slices multiply that.

Container images are the other consumer. Three EEs currently, roughly 6 GB.

df -h /var /tmp /var/log
du -sh /tmp/awx_* 2>/dev/null
sudo su - awx -c 'podman images'

VG has headroom for online extension:

sudo vgs
sudo lvextend -L +XG -r /dev/vg00/lv_var

`-r` resizes xfs in the same operation, online, no reboot.

1. Procedures

1. Restore a failed execution node

1. Confirm nothing running: `su – awx -c ‘podman ps’`
2. Resolve UID: `UID_AWX=$(id -u awx)`
3. Start user manager: `sudo systemctl start user@${UID_AWX}.service`
4. Verify `/run/user/${UID_AWX}` exists, owned by awx, mode 700
5. Restart receptor
6. Verify podman: `su – awx -c ‘podman info’` and `podman images`
7. Clear orphaned containers and `/tmp` debris
8. Check disk
9. Re enable in AAP UI, Administration > Instances
10. Run health check from the UI
11. Confirm capacity non zero: `awx-manage list_instances`
12. Smoke test, one host, forks 5, pinned to that node
13. Verify the Execution Node field on the job details

1. Resize an execution node

One node at a time. Never both.

1. Disable in AAP UI
2. Confirm no running jobs and no pending work units in `/var/lib/receptor`
3. `sudo systemctl stop receptor`
4. `sudo shutdown -h now`
5. In vCenter: adjust vCPU and RAM. Enable CPU Hot Add and Memory Hot Plug while powered off so the next resize needs no window
6. Power on
7. Verify: `free -g`, `nproc`, `lsmem`, user manager, runtime dir, receptor, podman, disk
8. Restart receptor after confirming the session is up, per Failure Mode 2
9. Re enable, health check, smoke test
10. Only then move to the second node

CPU topology note. CPU hot add only adds sockets. Building 1 socket with N cores makes future hot add impossible without a power off. Sockets keep hot add usable. At 32 GB vNUMA is not a factor.

1. Measure real per fork cost

Needed before sizing any new workflow.

1. Run the target playbook against a small host set, known fork count, slicing 1 so it lands on one node
2. Identify the node from the job details Execution Node field
3. After completion, on that node:

UID_AWX=$(id -u awx)
cat /sys/fs/cgroup/user.slice/user-${UID_AWX}.slice/memory.peak

4. Divide peak by fork count for incremental cost per fork
5. Size the full run from that, leaving headroom for OS, Defender, podman, and page cache

1. Sizing arithmetic

On a 32 GB exec node:

Fork count times measured per fork cost must fit in that. Windows runs roughly 1.5 to 2x the Linux per fork cost because every task spawns a PowerShell process on the target.

Forks should also respect cores. Above roughly 4 per core the scheduler thrashes and throughput drops. At 8 cores, 25 to 30 is the sane range.

1. Configuration reference

1. Job template settings, Linux patching

1. gather_subset

Not settable via extra vars. The implicit gathering task does not read a variable. Three valid places:

Play level, preferred:

- hosts: all
  gather_facts: true
  module_defaults:
    ansible.builtin.setup:
      gather_subset:
        - '!all'
        - '!min'
        - distribution
        - pkg_mgr

Or `ansible.cfg`:

[defaults]
gather_subset = !all,!min,distribution,pkg_mgr

Or an explicit setup task with `gather_facts: false` on the play.

Find what the role actually uses:

grep -rhoE 'ansible_[a-z0-9_]+' roles/ playbooks/ | sort -u

`hardware` is the expensive subset. It walks every block device and mount. Excluding it is most of the saving.

1. Capacity adjustment

Slider in Administration > Instances, per node. Interpolates between CPU derived capacity and memory derived capacity.

On exec nodes set toward CPU. Memory derived reads roughly 297 on 32 GB while CPU derived reads 32 on 8 cores. Leaving it memory weighted reproduces the misleading number that caused the incident.

It does not cap forks. Nothing does. Set forks explicitly on every template.

1. Defender exclusions

Applied to all nodes.

Exec nodes and controllers:

/var/lib/awx
/var/lib/containers
/var/lib/receptor

Gateway nodes:

/var/lib/ansible-automation-platform
/var/log/ansible-automation-platform

DB nodes, pre existing:

/var/lib/pgsql/   (scope=epp)

Verify:

sudo mdatp exclusion list

Job private data dirs under `/tmp` are dynamic and hard to exclude. Consider setting `AWX_ISOLATION_BASE_PATH` to `/var/lib/awx/tmp` so the existing exclusion covers them and `/tmp` stays scanned.

1. Open work

Priority order.

1. Capture memory peak from a measured run. Everything else about sizing depends on this number. Not yet done.

2. Add monitoring for the user manager. Both exec nodes. Two checks, resolved UID. This is the gap that let both nodes sit broken for 30 hours. Nothing native catches it.

3. Re enable scheduled jobs and workflows. All schedules were disabled during recovery. Freshservice snapshot, reaper, inventory source update, and sync_all_vms are off. The snapshot is stale.

Expect the first reaper run to show a larger Cat 5 delta than normal. The 20 percent spike halt may fire. That is correct behaviour. Verify the numbers before overriding.

4. Set capacity adjustment toward CPU on proden01 and proden02. Leave controllers alone.

5. Open the Red Hat case. Two findings:

- Exec node OOM kills the user manager, receptor survives and keeps heartbeating, node reports healthy while unable to start containers. Undetected for 30 hours.
- Receptor starts before the user manager on boot, creating a window where the node accepts work it cannot run.

Ask whether the exec node is expected to survive loss of the user manager, and whether adding `[email protected]` to the receptor drop in is supported.

6. Separate instance group for Windows patching. The planned 3000 host Windows workflow will squat on capacity for hours with reboots in the loop. Sharing `patching` with Linux work risks starvation.

Windows specific guidance when that workflow lands:

- Forks 15 to 20, lower than Linux. `MaxConcurrentOperationsPerUser` and `MaxShellsPerUser` on the targets throttle before the exec node does
- Slicing 8 to 12 for 3000 hosts
- Trim `gather_subset`, Windows fact gathering is expensive
- Use `win_reboot` with sane timeouts rather than blocking a fork for the full reboot

7. Decide on controllers in `default`. They are 6 cores and 15 GB, smaller than the exec nodes, and also carry gateway and Freshservice duty. Options: leave as is, remove from `default`, or convert to control only.

Control only requires migrating the Freshservice tools off prodac01 first. That means recreating the directories on the target node, copying `audit_ledger.jsonl` and `retire_baseline.json`, and repointing six job templates. Roughly half a day. Not a redo of the project.

8. Separate filesystems for `/var/lib/containers` and `/var/lib/receptor`. Currently both on a 100 GB `/var`. VG has about 30 GB free, not enough. Needs a new VMDK and another window per node. Low priority while utilisation is under 10 percent. Set an alert at 80 percent instead.

9. Controller sizing. prodac01 and prodac02 at 6 cores and 15 GB with EDA and agentic workloads planned against 8000 Linux and 3000 Windows hosts. Undersized. Not urgent now that patch work is pinned away from them.

1. Gotchas

1. awx UID differs per node. 1000 on proden01, 1002 on proden02. Never hardcode.

2. Capacity is memory derived and misleading. 136 on 15 GB, 297 on 32 GB, while CPU capacity is 16 and 32. Reading it as a fork budget caused the incident.

3. `sudo -u awx` is not a valid podman test. No PAM, no session, no runtime dir. Fails on healthy nodes. Use `su – awx` or `machinectl shell awx@`.

4. RSS double counts forked processes. Copy on write means shared pages are charged to every process. Use cgroup `memory.peak` or sum Pss.

5. `free` column versus `available` column. Free drops to 1 to 2 GB routinely while available stays healthy. Read available. Better, read `/proc/pressure/memory`.

6. `/api/v2/` 404s in AAP 2.6. Use `/api/controller/v2/`. Gateway health is at `/api/gateway/v1/ping/`.

7. Receptor being active proves nothing. It is a system service independent of the user session that podman needs. See Failure Mode 1.

8. Receptor starts before the user manager on boot. Restart receptor after any exec node reboot.

9. `[email protected]` is static with no Restart. Once killed it stays dead. Nothing brings it back automatically.

10. AAP does not fail jobs over between execution nodes. Slicing limits blast radius. There is no migration.

11. Job slicing 1 means no distribution. A second exec node contributes nothing to a single unsliced job.

12. `gather_subset` is not settable via extra vars. Play level `module_defaults`, `ansible.cfg`, or an explicit setup task.

13. Podman fallback path `/tmp/storage-run-` appearing means podman lost its runtime dir. Diagnostic signal for Failure Mode 1.

14. AAP keeps dispatching to a broken node. Nine job attempts hit proden01 in the 13 minutes after its user manager died. Failures are not enough to remove a node from the pool.

1. Incident log

1. Sep 2 2026, dual execution node OOM

02:49:59 proden01 system wide OOM. Linux patch job, 1000 hosts, forks 100, slicing 1, full fact gathering. Killed process 773118 at 1.55 GB resident. User manager PID 979 killed. `/run/user/1000` torn down.

02:53 to 03:02 Nine further job dispatches to proden01, all failed. `/tmp/storage-run-1000` created as podman fell back.

09:54:15 proden02 OOM, same job configuration. Killed process 3817737 at 1.2 GB resident. User manager PID 980 killed. Slice peak logged at 7.4 GB.

~10:00 Both exec nodes disabled manually in AAP.

Post 10:00 Patch job re dispatched to prodac01 via `default` group membership, forks 100, on a 6 core 15 GB controller also serving gateway and Freshservice duty.

Sep 3 Both exec nodes recovered. User managers restarted, receptor restarted, podman verified, `/tmp` cleared, filesystems extended, nodes resized to 8 cores and 32 GB, templates corrected, `patching` group created.

Root cause. Forks 100 with slicing 1 and full fact gathering on 4 core 15 GB nodes. Fork count chosen against the memory derived capacity figure of 136, which does not represent a fork budget.

Contributing. No Defender exclusions on AAP paths. No monitoring for user manager health. Controllers in `default` providing a silent fallback onto the control plane. Receptor heartbeat treated as a node health signal when it is not.

1. AGENTS.md

Rules for any coding agent working in this repo, aap-windows-patch-orchestrator.
These are always on. Read them before every task. Do not restate them back to me,
just follow them.

1. Mission
   Take this Windows patch orchestration project on Ansible Automation Platform 2.6
   from partial state to production ready. End to end pre and post patch workflow,
   real test coverage, stakeholder grade reporting. Ansible orchestrates and
   validates. SCCM, WSUS, and Patch My PC remain installers of record. You wrap
   them, you do not replace them.

1. Working style
   Short sentences. No hyphenated words. No emojis or icons in code, docs, or
   replies. State assumptions out loud. Push back when I am wrong. Reference Red Hat
   AAP and Ansible collection docs rather than guessing module behavior. Never
   invent module parameters, verify against the installed collection version.

1. Ground truth inputs, and how to treat them
   1. Design spec doc: intent and target state, not proof of what exists. Treat
   every claim as a hypothesis to verify against actual files.
   2. SCCM audit doc and WSUS audit doc: authoritative for server config,
   collection and group structure, deployment settings, reboot ownership, and
   maintenance windows. Use them to design the SCCM execution model and reboot
   cutover. Never contact prod SCCM or WSUS servers. Patch loop is client side.
   3. AAP topology reference: HA RPM cluster, F5 VIPs, gateway, Patroni Postgres
   control plane, receptor mesh, hosts file convention. Respect it.
   4. The repo itself: source of truth for what is actually built. Trust code over
   any doc. Some scaffold may have come from another assistant.

1. How you work
   Work from docs/DELIVERY-TRACKER.md, one scoped task at a time, one role or
   playbook or test per task. No big open ended runs. Feature branches, small
   reviewable commits, conventional commit messages. Run ansible lint, yaml lint,
   and pytest yourself and correct from real output, do not round trip through me
   for errors. Update PROJECT-STATUS.md and DELIVERY-TRACKER.md every session. When
   a change is destructive or a decision is ambiguous, stop and ask. When you
   assume, say so in the PR description.

1. Non negotiable design rules
   1. The JSON fact contract is the product. Every stage on every host emits one
   fact via roles/common/tasks/emit_fact.yml. All reporting materializes from
   that one contract. No parallel pipelines.
   2. Unified per host contract across mechanisms. An SCCM host and a WSUS host
   produce the same fact shape, so one report covers a mixed fleet.
   3. Postcheck is a health gate, not a scanner truth gate. It verifies WinRM
   restored, real reboot occurred, pending reboot cleared, services healthy, OS
   build and KB list captured. It does not call vulnerability scanners.
   4. Scanner reconciliation is deferred to Phase 3, T plus 24h to T plus 72h,
   because Rapid7, Tenable, and MDE re scan on 12 to 72 hour cycles. Calling
   them earlier emits stale data.
   5. Async multi phase. Phase 1 precheck, patch, reboot fire and forget, report.
   Phase 2 health gate, 5 to 15 minutes later, separate job. Phase 3 deferred
   compliance.
   6. SCCM is asynchronous. Triggering a deployment returns immediately with no
   pass or fail. Trigger then poll CCM_SoftwareUpdate EvaluationState on a
   bounded loop until each targeted update reaches terminal state, installed or
   error with code, then emit real pass or fail. Never treat SCCM like WSUS.
   7. On SCCM hosts do not use win_updates. SCCM installs, you poll to terminal,
   then reboot, then trigger client actions to reconcile, Software Updates Scan
   113, Deployment Evaluation 108, Hardware Inventory 101.
   8. On WSUS hosts use ansible.windows.win_updates for approved categories,
   capture per KB result inline, then report so WSUS status refreshes.
   9. Reboot must confirm terminal install state before firing on SCCM hosts.
   Rebooting mid install corrupts the patch and the report.
   10. Tag policy governs behavior. Appliance skips entirely. Hold_Power and
   Hold_Right_Size allow patch but suppress reboot. Standard allows both.
   Pending reboot before a cycle is a hard quarantine.
   11. dryrun predicts only, changes nothing, writes only cycle specific dryrun
   artifacts, never overwrites BI targets.
   12. Zero updates is a status, compliant already, not a blank cell. Validate SCCM
   zero against last scan time, WSUS zero against server sync timestamp.
   13. Secrets come from Vault AppRole at runtime, never written to disk, never
   committed. WinRM over HTTPS, kerberos for domain joined prod.
   14. Collections come from the private Automation Hub via the gateway VIP, pinned
   to versions Hub serves. EE builds from Hub. Preflight asserts the set.
   15. Do not over engineer. Simplest thing that works. No speculative abstraction.
   Measure before optimizing.

1. Reboot ownership, treat as a cutover
   Today SCCM auto reboots inside its maintenance window. Target is Ansible owns
   reboot so every reboot flows into the fact contract, Loki, and Redshift. This is
   a transfer, not an addition. If both reboot you get double reboots. Order, and
   document it in docs/CUTOVER.md, first stand up the AAP schedule that runs Phase 1
   then Phase 2 on the same Sunday windows, then set the SCCM deployment to suppress
   reboot, never the reverse. Hold tagged hosts never reboot regardless of owner.
   Maintenance windows are Sundays after the second Tuesday through the last Sunday
   of the month, 2 to 7 am Central.

1. Data and reporting boundaries
   Postgres, dedicated reporting instance, not the AAP Patroni control plane,
   authoritative for mutable cycle state and historical metrics. SMB share for
   immutable per cycle audit files plus overwritten fixed name files for BI and
   Redshift ingestion. Loki for real time stream only, never queried as a state
   store. The SMB xlsx column layout is a versioned contract with BI, changing it
   is a breaking change announced first.

1. Testing requirements
   Unit tests, pytest, for every python builder and the workflow decision model,
   covering tag policy, quarantine, cycle and batch id resolution, zero updates,
   and dry_run artifact naming. Molecule scenarios for roles with windows module
   calls mocked. A dry_run harness over a scoped mock inventory asserting no changes
   and correct fact emission. Connectivity smoke tests test_winrm.yml and
   test_smb.yml pass first. CI runs ansible lint, yaml lint, pytest, and a failing
   check blocks merge.

1. Safety guardrails
   Never run against prod inventory. Never call prod SCCM or WSUS servers. Use
   dry_run and the scoped pilot inventory only. The content lifecycle, second
   Tuesday sync, promote, pilot, deploy, stays exactly as is, do not automate or
   alter it. Fail one host, continue the fleet, never abort a batch on a single
   host error.

1. Definition of production ready
   All tests green in CI. EE builds and runs from Hub with pinned collections.
   Smoke tests pass on a real pilot host. A full dry_run cycle emits a correct
   unified report for a mixed SCCM and WSUS pilot. SCCM path proven to poll to
   terminal and report true pass or fail. Reboot ownership cutover documented and
   rehearsed. Phase 3 reconciliation produces a compliance delta. Runbook and
   cutover docs complete. No secret on disk, no direct prod SCCM or WSUS calls
   anywhere in the code.

Read AGENTS.md first and follow it. Then read the attached design spec doc, the
SCCM audit doc, the WSUS audit doc, and the AAP topology reference, and read the
actual repo. Attachments are intent, the repo is truth.

Your only job in this task is to look and plan. Write no feature code. Produce
two files and then stop for my review.

1. docs/GAP-ANALYSIS.md. Open with one honest paragraph on how far the repo is
from production ready. Then for every stage, role, playbook, and helper named
in the design doc, mark it works, stub or incomplete, or missing. List every
lint or build failure, undefined var, and any reference to a module or path
that is not present. Flag anything in the design doc that the code contradicts.

2. docs/DELIVERY-TRACKER.md. Weekly phases, each with goals, exit criteria, and a
status column set to not started. Seed with the phases below, then adjust
based on what the gap analysis actually found and say what you changed.

Do not write feature code until I approve both files.