• 09:54:15 proden02 runs out of memory. Kernel OOM killer fires repeatedly.
• 09:54:16 [email protected] dies. Peak consumption logged at 7.4 GB.
• ~10:00 Team member disables proden01 and proden02 in AAP.
• 17:32 Nodes still disabled, capacity 0. proden02 user manager still dead, /run/user/1002 missing.
• 18:35 A patch job is running on prodac01 instead, having fallen back to the control plane.

Node: proden02, 4 cores, 15 GB RAM, 3 GB swap.

Job: ~1000 hosts, forks 100, job slicing 1, gather_facts on.

Slicing 1 means one job on one node carrying all 1000 hosts. No distribution.

Memory budget on 15 GB:

What the workload actually used: systemd logged Consumed 7.4G memory peak on the user slice. That is parent playbook plus 100 forks plus podman. The parent alone was 1.2 GB resident at kill time.

7.4 GB inside the slice, on top of 3 to 4 GB of system overhead, on a 15 GB box with page cache also competing. That is the wall.

On the RSS number: the 410 MB per fork you saw is not additive. Copy on write means forks share most pages with the parent, and RSS counts shared pages fully in every process. Real incremental cost is more like 30 to 80 MB per fork. At 50 MB, 100 forks is 5 GB of genuine additional footprint. Consistent with the 7.4 GB slice peak.

AAP reported capacity=136 on every node. That number comes from the memory formula, roughly (MB minus 2048) divided by 100, which assumes 100 MB per fork. On 15 GB that gives ~133.

CPU derived capacity on the same node is cores times 4, so 16 on a 4 core exec node.

Default capacity adjustment weights memory, so the node advertises 136 while having 4 cores. Forks 100 looked reasonable against 136. It was never reasonable against 4 cores, and the 100 MB per fork assumption does not hold for a patch role gathering full facts.

1. Silent node breakage. The OOM killed PID 980, the systemd --user manager for UID 1002. [email protected] is static with no Restart directive, so it did not come back. /run/user/1002 is gone. Rootless podman needs that runtime dir.

Receptor kept running as a separate system service and kept heartbeating. So the node looked alive to the controller while being incapable of starting a container. Still in that state.

2. Control plane fallback. With both exec nodes at capacity 0, jobs targeting default had nowhere to go except prodac01 and prodac02, which are node_type=hybrid. The patch job landed on prodac01, which is also a gateway pool member behind the F5 and the sole node in fresh_sync_cp01.

• Every node is 15 GB. Red Hat's floor for an exec node is 16 GB, and that assumes modest concurrency. You have 8000 Linux and 2000 Windows hosts with EDA and agentic workloads planned.
• Defender has no exclusions for AAP paths. Your reference doc covers /var/lib/pgsql/ on the DB nodes only. /var/lib/awx/jobs, /var/lib/awx/projects, /var/lib/containers, and job private data dirs under /tmp are all being scanned during heavy artifact churn.
• Controllers are hybrid, which turns exec node loss into control plane risk.

Immediate

1. proden02: start [email protected], restart receptor, verify podman as awx.
2. Verify /run/user/1002 exists on proden01.
3. Re enable both nodes, run health check, confirm capacity non zero.

Template 4. Forks 100 to 25. 5. Slicing 1 to 4. Four jobs of 250 hosts distributed across both exec nodes. 6. Trim gather_subset to what the patch role needs. 7. Rerun against 50 hosts first, measure Pss, then scale up.

Structural 8. Defender exclusions for AAP paths on all nodes. 9. Add [email protected] and After= to the receptor drop in, so receptor cannot run without its session. 10. Move Freshservice tools off prodac01, then set controllers to control only. 11. Size to measured footprint. 15 GB is thin for this estate.

Earlier I told you to cancel the running job based on free swinging between 1 and 3 GB. Wrong column. Available was 8 GB, buff/cache was doing normal work. I should have asked for available before saying that.

for p in $(pgrep -f ansible-playbook); do
  awk -v p=$p '/^Pss:/{s+=$2} END{print p, s/1024" MB"}' /proc/$p/smaps_rollup
done

[root@proden02 ~]# systemctl status [email protected]
● [email protected] - User Manager for UID 1002
     Loaded: loaded (/usr/lib/systemd/system/[email protected]; static)
    Drop-In: /usr/lib/systemd/system/[email protected]
             └─10-login-barrier.conf
     Active: active (running) since Thu 2026-09-03 10:48:59 UTC; 11s ago
       Docs: man:[email protected](5)
   Main PID: 3904241 (systemd)
     Status: "Ready."
      Tasks: 2
     Memory: 6.0M (peak: 10.2M)
        CPU: 84ms
     CGroup: /user.slice/user-1002.slice/[email protected]
             └─init.scope
               ├─3904241 /usr/lib/systemd/systemd --user
               └─3904243 "(sd-pam)"

Sep 03 10:48:59 proden02.example.com systemd[3904241]: Reached target Timers.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Starting D-Bus User Message Bus Socket...
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Starting Create User's Volatile Files and Directories...
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Listening on D-Bus User Message Bus Socket.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Reached target Sockets.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Finished Create User's Volatile Files and Directories.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Reached target Basic System.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Reached target Main User Target.
Sep 03 10:48:59 proden02.example.com systemd[3904241]: Startup finished in 86ms.
Sep 03 10:48:59 proden02.example.com systemd[1]: Started User Manager for UID 1002.

[root@proden02 ~]# ls -ld /run/user/1002
drwx------. 3 awx awx 80 Sep  3 10:48 /run/user/1002

Node is disabled, nothing running on it. Safe to work on.

1.1 Confirm current state

systemctl status [email protected]
ls -ld /run/user/1002
id awx
ls -l /var/lib/systemd/linger/

Expect: service failed, runtime dir missing, awx is uid 1002, linger file present.

1.2 Start the user manager

sudo systemctl start [email protected]

1.3 Verify

systemctl status [email protected]
ls -ld /run/user/1002

Expect active running, and the directory present owned by awx with mode 700.

If it fails to start, stop here and send me the output.

1.4 Restart receptor

sudo systemctl restart receptor
sudo systemctl status receptor

1.5 Verify podman as awx with a real session

sudo su - awx -c 'podman info | head -30'
sudo su - awx -c 'podman images'

Expect podman info to return without error, and your EE image present:

aap-service.company.com/fresh-vc-sync/fresh-vc-sync-ee   1.0

Plus whatever EE your patch job uses. If images are missing, note it, they will pull on first run.

1.6 Check for leftover state from the OOM

sudo su - awx -c 'podman ps -a'
ls -la /tmp/awx_* 2>/dev/null
df -h /var/lib/awx /var/lib/containers /tmp /var/lib/receptor

Kill any orphaned containers from the crashed run:

sudo su - awx -c 'podman rm -f $(podman ps -aq)' 2>/dev/null

Clean orphaned private data dirs if /tmp is filling.

It did not OOM, but confirm it is actually healthy rather than assumed healthy.

systemctl status [email protected]
ls -ld /run/user/1002
systemctl status receptor
sudo su - awx -c 'podman info | head -20'
free -g
nproc
df -h /var/lib/awx /var/lib/containers /tmp /var/lib/receptor
journalctl -k --since "2026-09-01" | grep -iE "out of memory|oom-kill"

Expect everything healthy and no OOM history. This is your baseline for what a working node looks like.

3.1 From the UI

Administration > Instances. Toggle Enabled on proden01 and proden02. Then click the health check button on each.

3.2 Or from the API

curl -sk -u admin -X PATCH \
  -H "Content-Type: application/json" \
  -d '{"enabled": true}' \
  https://aap.example.com/api/v2/instances/<id>/

Get the ids first:

curl -sk -u admin https://aap.example.com/api/v2/instances/ \
  | python3 -m json.tool | grep -E '"id"|"hostname"|"enabled"|"errors"|"capacity"'

Read the errors field before enabling. If it says anything, tell me.

3.3 Confirm capacity

sudo awx-manage list_instances

Expect proden01 and proden02 no longer DISABLED, capacity roughly 133 to 136 each. If capacity is still 0 after a health check, stop and tell me.

Do not go straight to the patch job.

Create a throwaway job template:

• Playbook: a single debug task, or ping against one host
• Inventory: one host you can reach
• Instance group: leave default for now
• Forks: 5

Run it twice, once pinned to proden01, once to proden02. Confirm both complete and check the Execution Node field on the job details to prove it ran where you expected.

If proden02 fails here, podman is still broken and step 1 did not take.

This is the root cause fix.

5.1 Job template settings

• Forks: 100 to 25
• Job slicing: 1 to 4
• Leave privilege escalation and fact storage as is

5.2 Trim fact gathering

In the playbook, replace bare gather_facts: true with a subset. What you need depends on the role, but for most Linux patching:

- hosts: all
  gather_facts: true
  vars:
    ansible_facts_parallel: false
  tasks: []

Better, set it explicitly:

- hosts: all
  gather_facts: true
  module_defaults:
    ansible.builtin.setup:
      gather_subset:
        - '!all'
        - '!min'
        - distribution
        - pkg_mgr

Check what your role actually references before trimming. grep -r ansible_ roles/ will tell you.

5.3 Confirm fact cache is on

Settings > Job Settings. Fact storage you already have enabled. Set a cache timeout so repeat runs within the window skip gathering entirely.

6.1 Run against 50 hosts

Use --limit or a smaller inventory. Forks 25, slicing 1 for this test so it lands on one node and you can measure cleanly.

6.2 Measure while it runs

On whichever exec node it landed:

watch -n 15 'free -g; echo; cat /proc/pressure/memory'

And capture the real footprint:

cat /sys/fs/cgroup/user.slice/user-1002.slice/memory.current

After it finishes:

cat /sys/fs/cgroup/user.slice/user-1002.slice/memory.peak

6.3 Do the arithmetic

Peak divided by fork count gives you incremental per fork cost. That is the number you size everything else from, including the Windows workflow.

Send me the peak figure and I will sanity check the numbers I gave you earlier.

6.4 Then scale

250 hosts, slicing 4, both nodes. Measure again. Then full 1000.

All four nodes. Your reference doc has /var/lib/pgsql/ on the DB nodes only.

sudo mdatp exclusion folder add --path /var/lib/awx
sudo mdatp exclusion folder add --path /var/lib/containers
sudo mdatp exclusion folder add --path /var/lib/receptor
sudo mdatp exclusion folder add --path /var/lib/pgsql

Check your mdatp version for exact syntax, and whether you need --scope epp to match the pgsql exclusion pattern already in place.

Verify:

sudo mdatp exclusion list

Get security sign off before you do this. It is their call, not yours.

Replace the receptor unit change until Red Hat weighs in.

Add to whatever monitoring you run, per exec node:

systemctl is-active [email protected]
test -d /run/user/1002

Both must pass. Alert if either fails. That closes the nine hour blind spot without modifying vendor config.

Reproduction is clean:

• Exec node OOM kills [email protected], PID 980
• [email protected] is static with no Restart, does not recover
• receptor continues running as a system service, heartbeats normally
• Node reports healthy to controller, cannot start containers
• Undetected for nine hours

Ask specifically: is the exec node expected to survive loss of the user manager, and is adding [email protected] to the receptor drop in supported.

• Spec bump to 8 cores 32 GB, one node at a time, with hot add enabled
• Disk partitioning
• Instance group decision on whether controllers stay in default
• Separate instance group for Windows patching