Audience: Anyone reading this should understand the complete flow, data transformations, and decision points.

INPUT: Operator launches AAP workflow with inventory + batch_id
  ↓
PHASE 1 (same job, sequential stages):
  PreCheck → Patch → Report
  ↓
PHASE 2 (separate job, launched manually ~30 min after Phase 1):
  PostCheck Health-Gate
  ↓
PHASE 3 (scheduled independently, T+24-72h):
  Compliance Delta Verification

Input:

• target_hosts (or target_clusters, target_group) — inventory filter
• wpo_batch_id — correlation key for sliced inventory (optional; defaults to awx_workflow_job_id)
• wpo_env — "dev" or "prod"

Execution:

1. Inventory resolution: Get all Windows hosts matching the limit from vSphere dynamic inventory
2. For each host:
   • Fetch WinRM credentials from Vault (secret/infra/{env}/winrm)
   • Test WinRM connectivity
   • Query host state: OS version, pending reboot flags (Registry: CBS, WU, PFRO), disk space, services
   • Classify: Appliance? Powered off? Pending reboot already? Meets readiness threshold?
3. Quarantine decision:
   • If pending reboot detected → status: quarantined, reason: pending_reboot_before_cycle, end_host (skip to next)
   • If powered off → status: skipped, reason: powered_off
   • If Appliance tag → status: skipped, reason: appliance_skipped
   • Otherwise → status: ok, proceed to patch

Output per host:

{
  "inventory_hostname": "win10-prod-001",
  "ansible_host": "10.1.50.100",
  "precheck_status": "ok|quarantined|skipped",
  "precheck_reason": "pending_reboot_before_cycle|readiness_below_threshold|...",
  "precheck_free_pct": 45,
  "precheck_pending_reboot_before": true|false
}

Stored: Per-host stage fact → wpo_stage_facts['precheck']

Input: Host facts from PreCheck (via ansible_play_hosts_all)

Execution per host:

1. Skip if precheck_status != ok
2. Query WSUS/SCCM: "What patches are available?"
3. Install via Windows Update Agent or SCCM (based on patch_path var)
4. Record: KB IDs installed, reboot required flag
5. Error handling:
   • If WU transport error (e.g., HRESULT 0x80240438) → patch_status: failed, emit reason, end_host
   • If WSUS says "reboot required before install" → patch_status: skipped, reboot_required: true
   • Otherwise → patch_status: ok|changed

Output per host:

{
  "patch_status": "ok|changed|failed|skipped",
  "patch_attempted": true,
  "patch_changed": true,
  "patch_failure_reason": "wu_transport_error: ...",
  "needs_reboot": true|false,
  "kb_list": ["KB5123456", "KB5123457"],
  "patch_source": "wsus|sccm"
}

Stored: Per-host stage fact → wpo_stage_facts['patch']

Input: Accumulated wpo_stage_facts from all hosts + all prior stages

Execution:

1. Fetch SMB credentials from Vault (secret/infra/{env}/smb_report)
2. Build merged row per host:

   {
     "inventory_hostname": "win10-prod-001",
     "patch_cycle_id": "30804",
     "patch_batch_id": "2026-08-24-dev-winpatch",
     "precheck_status": "ok",
     "patch_status": "changed",
     "patch_changed": true,
     "needs_reboot": true,
     "kb_installed_count": 2,
     ...
   }

3. Build workflow summary:

   {
     "workflow_job_id": "30804",
     "patch_environment": "dev",
     "patch_batch_id": "2026-08-24-dev-winpatch",
     "generated_at": "2026-08-24T14:30:00Z",
     "hosts_targeted": 150,
     "hosts_precheck_ok": 145,
     "hosts_patch_ok": 140,
     "hosts_patch_changed": 125,
     "hosts_requiring_reboot": 120
   }

4. Write artifacts locally:
   • patch_report_latest_win.json (fixed name for BI ingest)
   • patch_report_dev_30804.json (cycle-specific copy for audit)
   • patch_report_latest_win.xlsx (spreadsheet: Summary, Per-Host, KBs, Failures, By-Cluster)
5. Push to SMB share: \\smb-share-drive\Collab\dev\Assets\aap-patch-report\
6. Push to Loki: 1 summary stream + N host streams (labels: job, environment, batch_id, workflow_job_id)
7. Emit metrics to Prometheus

Key insight: Report shows which hosts need reboot, but does NOT reboot them yet if using SCCM/WSUS scheduler.

When: Launched manually ~30 min after Phase 1 (time for SCCM/WSUS reboot window)

Execution per host:

1. Skip if precheck_status == quarantined (reason: pending reboot before cycle)
2. Test WinRM liveness (post-reboot)
3. Verify pending-reboot flags cleared (Registry: CBS, WU, PFRO)
4. Verify OS actually rebooted (check LastBootUpTime, uptime_seconds)
5. Verify required services running (WinRM, LanmanServer, W32Time)
6. Record: OS version, build, KB list

Output per host:

{
  "postcheck_status": "ok|failed|skipped",
  "postcheck_reason": "winrm_failed|services_down|...",
  "postcheck_last_boot_at": "2026-08-24T14:15:00Z",
  "postcheck_uptime_seconds": 900,
  "postcheck_pending_reboot_after": false,
  "postcheck_kb_list": ["KB5123456", "KB5123457"]
}

Stored: Per-host stage fact → wpo_stage_facts['postcheck'] (merged with prior stages)

When: Scheduled independently, T+24-72h after Phase 1

Purpose: Verify patch state vs. current MDE/security posture

Input: wpo_cycle_id from Phase 1 (passed as extra var)

Execution:

1. Read Phase 1 report from SMB: patch_report_dev_<cycle_id>.json
2. For each host, query MDE Custom Attributes (stored in vSphere):
   • vm_attr_defender_health — current MDE status
   • vm_attr_defender_onboard — onboarded state
3. Cross-check: "Was this host patched in cycle 30804?" vs. "Does MDE show patches installed?"
4. Flag delta: Patched in AAP but MDE says unpatched? → investigate
5. Build compliance delta xlsx:
   • Summary: Total hosts, matched, mismatched
   • Per-host: Patch state vs. MDE state, delta reason

Output:

• win_compliance_delta_dev.json + win_compliance_delta_dev_30804.json (cycle-specific)
• win_compliance_delta_dev.xlsx (spreadsheet)

┌─────────────────────────────────────────────────────────────────┐
│ Vault (Source of Secrets)                                       │
│  ├─ secret/infra/dev/winrm → {username, password}              │
│  ├─ secret/infra/dev/smb_report → {server, share, user, pwd}   │
│  └─ secret/infra/dev/rapid7 → {client_cert}                    │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ vSphere Dynamic Inventory (Source of Host Metadata)             │
│  ├─ vm_cluster, vm_power_state, vm_guest_os                    │
│  ├─ vm_tag_appliance_present, vm_tags_raw                      │
│  ├─ vm_attr_defender_health, vm_attr_sccm, vm_attr_wsus       │
│  └─ vm_guest_ip → ansible_host resolution                       │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ AAP Execution (playbooks/site.yml or phase1.yml)                │
│                                                                  │
│  PHASE 1 ────────────────────────────────────────────────────   │
│  PreCheck Role                                                   │
│    ├─ Input: Vault creds, vSphere inventory                    │
│    └─ Output: wpo_stage_facts[precheck] per host               │
│           ↓                                                      │
│  Patch Role                                                      │
│    ├─ Input: wpo_stage_facts[precheck] + Vault creds           │
│    └─ Output: wpo_stage_facts[patch] per host                  │
│           ↓                                                      │
│  Report Role                                                     │
│    ├─ Input: wpo_stage_facts[precheck] + wpo_stage_facts[patch]│
│    └─ Output: xlsx + JSON files                                 │
│           ↓                                                      │
│  (Credentials fetched at runtime from Vault, not stored in AAP) │
└─────────────────────────────────────────────────────────────────┘
                              ↓
┌─────────────────────────────────────────────────────────────────┐
│ Artifact Delivery (Report Role)                                 │
│                                                                  │
│  1. SMB Share: win_patch_compliance_report_dev.json/.xlsx      │
│     ↓ (consumed by BI/Redshift)                                 │
│  2. Loki: HTTP POST streams (labels: batch_id, workflow_job_id)│
│     ↓ (visualized in Grafana)                                   │
│  3. Prometheus: Metrics (patch_workflow_hosts_patched, etc.)    │
│     ↓ (time-series alerting)                                    │
└─────────────────────────────────────────────────────────────────┘

Host Ready for Patch?
│
├─ Powered off? → SKIP (reason: powered_off)
│
├─ Appliance tag present? → QUARANTINE (reason: appliance_skipped)
│
├─ Pending reboot flags set?
│  (Registry: HKLM:\System\CurrentControlSet\Control\Session Manager\PendingFileRenameOperations
│   or COM: Microsoft.Update.SystemInfo.RebootRequired)
│  YES → QUARANTINE (reason: pending_reboot_before_cycle)
│
├─ Free disk < threshold? → QUARANTINE (reason: readiness_below_threshold)
│
└─ Otherwise → PATCH OK

Patch Outcome?
│
├─ Precheck status != ok? → SKIP
│
├─ WSUS reports: "Reboot required before install"? 
│  YES → Skip patch, set needs_reboot=true, let SCCM/WSUS handle reboot
│
├─ WU transport error? (e.g., HRESULT 0x80240438)
│  YES → FAIL, emit reason
│
├─ Install succeeded?
│  YES → CHANGED (if KB count > 0) or OK (0 KBs)
│
└─ Record: KB list, needs_reboot flag

Host Ready for PostCheck?
│
├─ Precheck was QUARANTINED? → SKIP (reason: skipped_due_to_precheck_quarantine)
│
├─ WinRM unreachable? → FAILED (reason: winrm_unreachable_post_reboot)
│
├─ Host actually rebooted?
│  (Check: LastBootUpTime > Phase 1 start time)
│  NO → FAILED (reason: reboot_not_detected)
│
├─ Pending reboot flags still set? → FAILED (reason: pending_reboot_after)
│
├─ Required services running? → FAILED (reason: service_down: <svc_name>)
│
└─ Otherwise → OK

1. AAP Job Template has ONE attached credential: Vault AppRole
2. AppRole auth (idempotent, run-once):

   POST https://vault.example.com/auth/approle/login
   Body: {"role_id": "$VAULT_APPROLE_ROLE_ID", "secret_id": "$VAULT_APPROLE_SECRET_ID"}
   Response: {auth: {client_token: "hvs.CAESIAg..."}}
   

3. Fetch WinRM creds:

   GET https://vault.example.com/data/secret/infra/dev/winrm
   Header: X-Vault-Token: hvs.CAESIAg...
   Response: {data: {data: {username: "DOMAIN\\svc_aaporch", password: "***"}}}
   

4. Set Ansible vars per host:

   ansible_user: DOMAIN\svc_aaporch
   ansible_password: ***
   ansible_connection: winrm
   ansible_port: 5985 (or 5986 for HTTPS)
   

5. WinRM connection: Ansible uses set vars to connect to ansible_host (resolved from VM IP/hostname)

Key: No SMB or Machine credentials attached to AAP. All fetched at runtime from Vault.

\\smb-share-drive\Collab\dev\Assets\aap-patch-report\
  ├─ win_patch_compliance_report_dev.json    (always this name; Redshift polls this)
  ├─ win_patch_compliance_report_dev.xlsx
  ├─ win_compliance_delta_dev.json           (Phase 3 compliance; Redshift polls this)
  └─ win_compliance_delta_dev.xlsx

  ├─ win_patch_compliance_report_dev_30804.json
  ├─ win_patch_compliance_report_dev_30804.xlsx
  ├─ win_compliance_delta_dev_30804.json
  └─ win_compliance_delta_dev_30804.xlsx

Purpose: BI consumes fixed names; auditors/compliance read cycle copies.

Each AAP job pushes to Loki after Phase 1 Report completes.

Stream labels (immutable query keys):

job: aap-dev-windows-patch
environment: dev
batch_id: 2026-08-24-dev-winpatch        # Correlation key for sliced inventory
workflow_job_id: 30804                   # Phase 1 workflow job ID
event: patch_host_result | patch_workflow_summary
hostname: win10-prod-001                 # Only on patch_host_result
os: WIN2022                              # Only on patch_host_result
platform: windows

Log line (JSON value):

{
  "event": "patch_host_result",
  "workflow_job_id": "30804",
  "patch_cycle_id": "30804",
  "patch_batch_id": "2026-08-24-dev-winpatch",
  "patch_environment": "dev",
  "inventory_hostname": "win10-prod-001",
  "precheck_status": "ok",
  "patch_status": "changed",
  "needs_reboot": true,
  "postcheck_status": "not_checked",
  ...
}

Grafana query example:

{job="aap-dev-windows-patch", batch_id="2026-08-24-dev-winpatch"}
| json
| patch_failed="true" or needs_reboot="true"

1. Launch Phase 1:

   • Template: dev-win-site-full-cycle or dev-win-patch (depending on your naming)
   • Inventory: vSphere-Dynamic-dev
   • Limit: cluster-A or dev-win-patch-test or custom group
   • Extra vars:

     wpo_env: dev
     wpo_batch_id: 2026-08-24-dev-winpatch
     winrm_dns_suffix: dev.example.com
     wpo_loki_push_enabled: true

   • Launch → Phase 1 runs (PreCheck → Patch → Report)

2. Wait: ~5-30 min for SCCM/WSUS reboot window

3. Check SMB: Operator verifies win_patch_compliance_report_dev.json exists on share

4. Launch Phase 2 (separate template):

   • Template: dev-win-postcheck
   • Same limit as Phase 1
   • Extra vars:

     wpo_env: dev
     wpo_batch_id: 2026-08-24-dev-winpatch

   • Launch → Phase 2 runs (PostCheck only)

5. Monitor Grafana: Dashboard AAP Windows Patch Workflow (Loki)

   • Filter by $batch_id to see all 3 phases grouped
   • View failures panel (filtered by patch_failed, reboot_status, etc.)

6. Schedule Phase 3 (compliance delta):

   • Launch T+24-72h after Phase 1
   • Pass wpo_cycle_id: 30804 (Phase 1 job ID)
   • Runs in background; outputs win_compliance_delta_dev_30804.json

1. Vault is the only credential store — No SMB, no Machine creds in AAP
2. Phase separation — Each phase can run independently; data flows via Loki + SMB artifacts
3. Fire-and-forget reboot — Reboot request sent, AAP doesn't wait; Phase 2 health-gates later
4. Batch correlation — batch_id groups sliced inventory jobs in Grafana
5. Pending-reboot quarantine — Blocks patching on half-updated hosts; recovery is manual + deliberate
6. Deferred compliance — Scanner lag doesn't block Phase 1; Phase 3 runs later for truth
7. Async multiple collection groups — Each SCCM deployment is independent; AAP just reports patches + reboot need

Version: 2026-08-24
Status: Current design (Phase 1-2 green, Phase 3 pending compliance truth validation)