AAP Windows Patch Orchestrator ‐ Review - SS67/project-docs GitHub Wiki

Blocking — before first cycle

Infrastructure

  • Resize both execution nodes: 4→8 vCPU, 16→32GB
  • Verify reported capacity in AAP UI after service restart (not just after boot)
  • Leave capacity_adjustment at 0.5 for now
  • Set per-slice forks from measured capacity (~330 total ÷ 15 slices ≈ 20); confirm all 15 slices enter running, not pending, at launch
  • Check vSphere CPU ready time and any leftover memory limit/reservation from the 16GB config

All accepted. Three follow-on implications worth pinning down:

Hold_Power / pending-reboot — you're right, and it goes further. Blanket exemption is unsafe; patching on top of an unknown pending-reboot state can land you mid-CBS-transaction. The workable distinction is provenance: a pending reboot you created last cycle and recorded (known KB set, known deferral) is a different object from one of unknown origin, and only the former is a candidate for proceeding. But even that needs a bounded depth — stacking two or three cycles of unactivated updates on one host is where servicing genuinely breaks. I'd allow one stacked cycle, then hard-stop and escalate for a reboot window rather than continuing to patch.

The larger point underneath: a Hold_Power host is never actually compliant. The binaries are staged, the fixes aren't live. Its status should read patched_pending_activation, not compliant, and Rapid7 will correctly keep flagging it — that's not a false delta, it's the truth. Under a 30-day critical SLA, an indefinitely-deferred host is an SLA breach with a tag on it. Worth making the age tracking visible enough that "Hold_Power" doesn't become a permanent parking spot.

Deadline — agreed, and the gap is wider than timing. Fan-in must classify by absence of artifact, not by receiving a failure signal, since a dead executor sends nothing. The consequence to plan for: a slice that died may have left hosts mid-install or mid-reboot, and their true state is unknown — not failed, not complete. Those need explicit reconciliation at the start of the next cycle (query actual patch state, don't assume), rather than being treated as ordinary retry candidates.

SCCM cohort validation — one thing to get right. Test the end state, not a mixed one: disable the schedule for the validation collection specifically, so that cohort has Ansible as sole driver. If the SCCM schedule stays live while Ansible also patches, you're testing a race, and whatever you observe won't predict the rollout.

Nothing further from me on the to-do list. The six open questions stand as written, with dependency pairs and rollback still the two I'd close before the first consolidated run — everything else can be answered from cycle data.

High value — first few cycles

  • Config-validity checks moved into precheck: WUServer/WUStatusServer, SCCM client health, assigned collection, last successful detection (parked on your plan)
  • Zero-KB result only counts green after fresh successful detection
  • Lead slice at 8:00, fleet at 8:45, with a preflight gate reading the lead slice outcome
  • Per-stage timestamps in stage facts — measure where the 45 minutes actually goes
  • Pre-stage content Monday (DP pre-download or download-only pass) and pre-run detection Monday evening
  • Drop hostname from Loki stream labels; query via | json
  • Disk-space remediation (WU cache / DISM) attempt before quarantining
  • Gate compliance reconciliation on Rapid7 scan freshness rather than fixed T+48h
  • Tag patch_ring on all hosts now, even though nothing reads it yet
  • Verify no_log on the Vault fetch and the ansible_password set_fact
  • Fix naming drift in the design doc (three different report filenames) and Appliance skip-vs-quarantine inconsistency
  • Monitor control node / database during first full run — "finalizing" lag is the symptom

Failure loop

  • Persist per-host cycle outcomes; compute consecutive_missed_cycles from Redshift
  • Auto-generate tickets at a threshold (3+ misses) to VM owner with reason code and webhook link — no human triage step
  • Route the manual workflow by reason code, not by "failed"
  • Headline metric on Grafana = aging distribution, not per-cycle failure count

Decisions to write down

  • No snapshots in dev; recovery is rebuild-from-template
  • SCCM/WSUS are content sources only; AAP owns install and reboot timing
  • Canary ring deferred 2–3 months, revisit after cycle data
  • Rapid7 3×/week dev, 30-day critical SLA

Still open

  1. Dependency pairs. Tags handle "don't reboot this host," but not "don't reboot these two together." Are all DC/SQL/HA members Appliance-tagged, or could a dependent pair land in the same batch?
  2. Standalone local admin accounts — per-host and rotated, or a shared static credential across ~600 workgroup VMs?
  3. Webhook idempotency — is it re-runnable for drift (GPO reverting WinRM, image refresh), and can the patch cycle trigger re-onboarding on WinRM failure, or only report it?
  4. Exclusion mechanism — how a host is legitimately taken out of scope, and does the exclusion expire?
  5. Service account durability — excluded from GPOs that strip WinRM config or lock the account out, across 4000 hosts over months?
  6. Rollback — what you do if a cumulative update breaks broadly. win_updates uninstall is unreliable for CUs; practical answer is DISM or rebuild. Worth knowing before the night you need it.

Of these, 1 and 6 are the ones I'd want answered before the first consolidated run — the rest can settle over the first few cycles.