MLM ‐ Saltstack - SS67/project-docs GitHub Wiki

SUSE MLM 5.1 Design: Single Server vs HA

Sep 29, 2026 · @sujay

Purpose and scope

Decision this doc supports: run SUSE Multi-Linux Manager (MLM) 5.1 as a single tuned server on vSphere HA with proxies, or as a Pacemaker active passive pair using the SLE High Availability Extension. Recommendation is the single server. Details below so the SUSE meeting can confirm or challenge it.

In scope: MLM 5.1 server and proxy topology in one datacenter, roughly 10,000 Salt minions, storage on Pure FlashArray, backup and restore, SKU impact, the boundary between MLM and AAP. Out of scope: Windows hosts (stay on AAP), Liberty Linux migration mechanics per host, EDA design.

Assumptions carried: MLM 5.1 on SL Micro 6.1, embedded Postgres, RHEL hosts move to SUSE Liberty Linux, outage tolerance for the MLM server is minutes, one datacenter.

Background

Red Hat Satellite lapses and Linux content management moves to MLM. Today every patch workflow runs through AAP inventory, so all 8,000 Linux hosts count as AAP managed nodes and drive the AAP subscription. Moving host based orchestration to Salt minions under MLM removes most of those hosts from the AAP count.

RHEL hosts move to SUSE Liberty Linux for content and support. MLM manages RHEL as a client, but it does not supply RHEL repos. Without Liberty or a retained RHEL subscription there is nothing to patch from.

Layer Owner after migration Notes
OS content, channels, CLM MLM Liberty and SLES channels, clones, lifecycle stages
Patching, package state, config state MLM via Salt Salt states, system groups, maintenance windows
Host inventory of record for Linux MLM Salt grains and hardware profiles
Cross system orchestration AAP vSphere, Freshservice, Vault, Windows, approvals
Windows hosts (2,000) AAP Unchanged
Event driven triggers AAP EDA Can call MLM API to run Salt jobs

AAP keeps a small inventory: the MLM server and proxies, and whatever Linux hosts still need Ansible content that has no Salt equivalent. Target is well under 1,000 Linux managed nodes in AAP. MLM 5.1 also ships Ansible control node integration, so the reverse path exists if a playbook is easier than a state.

Migration rule for each AAP job template: targets a Linux host in inventory, it becomes a Salt state or MLM action. Targets an API (vCenter, Freshservice, Vault, MLM itself), it stays in AAP. Windows stays in AAP. Target AAP footprint after cutover: 2,000 Windows, the MLM server and 4 proxies, and under 200 Linux hosts that still need Ansible only content. Everything else leaves AAP inventory.

Reframing: HA vs mono is an MLM server question, not a Salt question

In MLM the Salt master runs inside the server container. There is no supported way to run a separate Salt master pair, Salt multi master, or a syndic tree under MLM. Every minion talks to the MLM server directly or through an MLM proxy that forwards to that one server. So the choice is not two Salt masters versus one. It is one MLM server versus one MLM server that can fail over to a standby node.

SUSE documents three ways to keep a 5.1 server available:

  1. Single server, hypervisor HA, mgradm backup and restore for disaster recovery. This is what SUSE ships tooling for. Their product page says if the server fails, spin up another container (SUSE).
  2. Active passive Pacemaker cluster with shared storage using SLE HA Extension. Possible, not documented by SUSE for 5.x, and the container volumes must all live on the shared disk. Confirm support posture in the meeting.
  3. Hub with peripheral servers. Scale out for tens of thousands of clients across servers. Not HA for a single server, and each peripheral is itself a single point.

Proxies are the scale lever, not a second master. SUSE says clients should never attach directly to the server in production and one proxy suits 500 to 1,000 clients (hardware requirements). One tuned server with enough proxies is the documented ceiling at 10,000 clients (install guide).

Rejected: standalone Salt multi master HA pointed at MLM. MLM must be the master. A minion cannot serve two masters with different pillar and state trees without conflicts, and MLM has no way to register or track a foreign master. The only workable form is open source Salt multi master with MLM demoted to a repo mirror. That drops CLM, action scheduling, maintenance windows, compliance reporting, and SUSE support for orchestration, while keeping the MLM subscription cost. Worse than Option A on every axis except sub minute failover of the master, which Option B also gives if SUSE supports it.

Salt infrastructure design

Salt 3006 LTS, one master inside the MLM server container, salt-broker on each proxy, standard minions on every host. No salt-ssh except for bootstrap fallback. MLM owns the master config, key acceptance, and generated pillar. We own custom states, custom pillar, and reactor rules in git.

Components and where they live:

Component Location Managed by
salt-master MLM server container, ports 4505 and 4506 MLM, config in etc-salt volume
salt-broker Each proxy container, minions connect here mgrpxy
salt-minion Every Linux host, master set to its proxy FQDN Bootstrap script, activation key
Master config overrides /etc/salt/master.d/custom.conf inside container Us, via mgrctl term
Custom states /srv/salt inside container, volume srv-salt Git via gitfs, one repo
Custom pillar /srv/pillar inside container, volume srv-pillar Git via ext_pillar git
MLM generated pillar /srv/susemanager/pillar_data MLM, do not edit
Reactor rules /etc/salt/master.d/reactor.conf Us, git
Formulas with forms /srv/formula_metadata, /usr/share/salt-formulas RPM or git, UI driven

Environments and targeting:

  • Salt fileserver environments: base only. Lifecycle staging is done by MLM CLM channels and system groups, not by Salt environments. Two staging systems fighting each other is the classic mistake.
  • Targeting from MLM: system groups map to Salt grains susemanager:groups. Use groups for patch waves, grains for OS facts, pillar for site data. Avoid glob targeting on hostnames.
  • Nodegroups in master config only for ad hoc CLI work from mgrctl term, never in scheduled actions.
  • Highstate content: MLM assigns states per system or group in the UI. Our git repo provides the state tree, MLM decides who gets what. Keep the repo as a flat set of formulas, one directory per concern.

Key and identity lifecycle:

  • Minion id = FQDN. Bootstrap through the proxy with an activation key per environment and OS, for example 1-liberty9-prod. Key auto accepted by the activation key, no manual salt-key -a.
  • Rebuilt VM with the same name: AAP deletes the old system in MLM first (system.deleteSystems), then bootstraps. Otherwise the new key is rejected as a duplicate.
  • Retired VM: VmRemovedEvent in EDA triggers MLM system.deleteSystems, which also drops the key. This is the same guarded retire pattern as the Freshservice reaper, reuse the age check.

Execution safety on 10,000 minions:

  • java.salt_batch_size=200 in rhn.conf so MLM sends actions in batches, not to all at once.
  • worker_threads=24, timeout=60, gather_job_timeout=20 in master config, tune after 1,000.
  • Presence ping before every action, MLM does this by default, keep java.salt_presence_ping_timeout=4.
  • No salt '*' state.apply from the CLI in production. Everything scheduled through MLM so the action log, batching, and maintenance windows apply.

Event bus and reactor, the hook into AAP:

flowchart LR
    MIN[Minion] -->|job return| BUS[Salt event bus\non master]
    BUS --> MLM[MLM action tracker\naction status in DB]
    BUS --> REACT[Reactor\nsalt/job/*/ret/*]
    REACT -->|http.query| AAPAPI[AAP API\nlaunch workflow]
    MLM -->|XMLRPC schedule.listCompletedActions| AAP[AAP polling job]
    AAPAPI --> AAP
Loading

Pick one direction and keep it. Recommended: reactor is only for failures (retcode != 0 on patch actions posts to AAP), everything else AAP pulls from the MLM API at the end of the window. Push on failure, pull for status. That avoids 10,000 reactor calls per patch cycle.

Minion config baseline, pushed by a state after bootstrap:

# /etc/salt/minion.d/site.conf
master: proxy01.example.com
master_alive_interval: 60
auth_timeout: 30
random_reauth_delay: 120
recon_default: 1000
recon_max: 60000
recon_randomize: True
grains_cache: True
grains_cache_expiration: 300

The recon and randomize lines matter. After a server or proxy restart 2,500 minions reconnecting at once is a thundering herd on the broker. Spread them.

Master overrides inside the container:

# /etc/salt/master.d/custom.conf
worker_threads: 24
timeout: 60
gather_job_timeout: 20
fileserver_backend:
  - roots
  - gitfs
gitfs_remotes:
  - https://git.example.com/infra/salt-states.git:
    - base: main
gitfs_update_interval: 300
ext_pillar:
  - git:
    - main https://git.example.com/infra/salt-pillar.git
reactor:
  - 'salt/job/*/ret/*':
    - /srv/reactor/notify_aap_on_fail.sls

Apply with mgrctl exec -- systemctl restart salt-master. Keep this file in the same git repo and treat it as a MLM upgrade checklist item, upgrades can reset master.d.

Target architecture and workflows

One MLM server on vSphere, four proxies fronting all minions, AAP calling MLM over its XMLRPC API for cross system work. Mermaid sources below are copy ready for the repo docs.

Topology:

flowchart TB
    SCC[SUSE Customer Center] -->|mgr-sync| MLM[MLM Server 5.1\nSalt master, Postgres, Tomcat, Apache]
    MLM --> P1[Proxy 01]
    MLM --> P2[Proxy 02]
    MLM --> P3[Proxy 03]
    MLM --> P4[Proxy 04]
    P1 --> M1[Minions 2500]
    P2 --> M2[Minions 2500]
    P3 --> M3[Minions 2500]
    P4 --> M4[Minions 2500]
    AAP[AAP Controller] -->|XMLRPC API| MLM
    EDA[AAP EDA] -->|vCenter events| AAP
    VC[vCenter] --> EDA
    MLM -->|mgradm backup| BK[(Pure NFS backup target)]
    VS[vSphere HA + DRS] -.restarts VM on host loss.-> MLM
Loading

Content flow (channel lifecycle):

flowchart LR
    SCC[SCC vendor channels] --> SYNC[mgr-sync daily]
    SYNC --> VENDOR[Vendor channel\nLiberty 9, SLES 15]
    VENDOR --> CLM[Content Lifecycle Project]
    CLM --> DEV[dev stage]
    DEV -->|promote| TEST[test stage]
    TEST -->|promote| PROD[prod stage]
    DEV --> G1[System group dev]
    TEST --> G2[System group test]
    PROD --> G3[System group prod]
    PROD --> PX[Proxy squid cache]
    PX --> MIN[Minions pull packages]
Loading

Patch workflow, MLM native, AAP only for approval and CMDB:

flowchart TD
    CAL[Maintenance window\nMLM calendar] --> PROMO[CLM promote to prod]
    PROMO --> ACT[Schedule patch action\nsystem group + window]
    ACT --> SALT[Salt master pushes job\nvia proxy]
    SALT --> MIN[Minion applies patches]
    MIN --> REBOOT{Reboot required?}
    REBOOT -->|Yes| RB[Reboot action, same window]
    REBOOT -->|No| DONE[Action complete]
    RB --> DONE
    DONE --> HOOK[MLM API or Salt event]
    HOOK --> AAP[AAP workflow]
    AAP --> FS[Freshservice ticket update]
    AAP --> VC[vSphere snapshot cleanup]
Loading

AAP to MLM boundary at runtime:

sequenceDiagram
    participant EDA as AAP EDA
    participant AAP as AAP Controller
    participant MLM as MLM API
    participant SALT as Salt master
    participant MIN as Minion
    EDA->>AAP: VmCreatedEvent
    AAP->>MLM: system.bootstrap (uses bootstrap script via proxy)
    MLM->>SALT: accept key, apply highstate
    SALT->>MIN: state.apply
    MIN-->>SALT: return
    SALT-->>MLM: action result
    MLM-->>AAP: action id status
    AAP->>AAP: update Freshservice, close workflow
Loading

Reading: AAP never touches the Linux host directly after bootstrap. It asks MLM to do it. That is what keeps the AAP managed node count down.

Option A: single server on vSphere HA, proxies, mgradm backup

One MLM VM on SL Micro 6.1, container volumes on a dedicated Pure LUN, vSphere HA and DRS restart the VM on ESXi host loss, nightly mgradm backup to Pure NFS, plus a VM level snapshot before every MLM upgrade. Recovery time on host loss is VM boot plus container start, about 3 to 6 minutes. Recovery from data corruption is a restore, 30 to 90 minutes depending on channel volume size.

What fails and what happens:

Failure Effect on minions Recovery
ESXi host dies Minions reconnect to proxies, proxies buffer until server returns vSphere HA restart, automatic
MLM VM OS corrupt Same, no new actions until restored Revert VM snapshot or rebuild host and mgradm backup restore
Postgres corrupt Scheduled actions pause mgradm backup db restore from WAL archive
Pure LUN lost Full outage Rebuild from mgradm backup on NFS, re sync channels
One proxy dies Only its 2,500 minions lose contact Repoint those minions to a spare proxy or restore proxy VM

During a server outage patches already scheduled do not run and new registrations wait. Running minions keep running. Salt returns from in flight jobs are lost if the outage crosses the job. Nothing on the managed hosts breaks.

flowchart LR
    subgraph vSphere cluster
        ESX1[ESXi 01] --- ESX2[ESXi 02] --- ESX3[ESXi 03]
        MLM[MLM VM] -.HA restart.-> ESX2
    end
    MLM --> LUN[(Pure LUN\ncontainer volumes)]
    MLM -->|nightly| NFS[(Pure NFS\nmgradm backup)]
    MLM --> PX[Proxies 01 to 04]
    PX --> MIN[10k minions]
Loading

Option B: active passive with SLE HA Extension

Two SL Micro or SLES 15 SP7 nodes, Pacemaker and Corosync, one shared Pure LUN holding all container volumes, a floating IP that is the MLM FQDN, SBD fencing on a small shared disk. Only one node runs the containers at a time. Failover is 30 to 90 seconds including container start.

The catch: SUSE does not publish a Pacemaker procedure for the containerized 5.x server. Uyuni upstream had one for 4.x RPM installs. Building it means writing resource agents around mgradm start and mgradm stop, mounting the volume root on failover, and owning that config through every MLM upgrade. Every support case starts with SUSE asking if it reproduces on a standalone server.

What it protects against: node OS failure, VM crash, ESXi host loss. What it does not protect against: Postgres corruption, a bad upgrade, an operator mistake, or loss of the shared LUN. Those are the failures that actually cost hours, and they hit both nodes at once because the data is shared.

flowchart LR
    VIP[Floating IP\nmlm.example.com] --> N1
    subgraph Pacemaker cluster
        N1[Node 01 active\npodman containers running]
        N2[Node 02 passive\npodman idle]
        N1 <-->|corosync| N2
    end
    N1 --> LUN[(Shared Pure LUN\n/var/lib/containers/storage/volumes)]
    N2 -.mounts on failover.-> LUN
    N1 --> SBD[(SBD fencing disk)]
    N2 --> SBD
    N1 --> PX[Proxies]
Loading

Cost: SLE HA Extension on both nodes, a second VM of full server size sitting idle, shared disk multi writer config on vSphere, and cluster operations skill on a team that otherwise has none of it in this stack.

Option C: Hub with peripheral servers

One Hub server holds channels and pushes them to peripheral servers with Hub Online Sync (new in 5.1). Minions register to peripherals. This is for tens of thousands of clients or several sites with their own ops teams. At 10,000 clients in one datacenter it doubles server count and adds sync lag for no availability gain. Park it. Revisit if Windows ever moves to Salt or a second datacenter comes online.

Server specs and storage

Sizing for 10,000 clients, about 12 channels (Liberty 8, Liberty 9, SLES 15 SP6 and SP7 plus clones per CLM stage), one datacenter. SUSE minimums are 4 cores and 16 GB, and 64 GB or more for thousands of clients (hardware requirements). Numbers below are the production target, not the minimum.

Role Count vCPU RAM GB OS disk GB Data disk GB Notes
MLM server (Option A) 1 16 64 100 1,000 channel volume + 300 Postgres volume Two separate Pure LUNs via mgr-storage-server
MLM server nodes (Option B) 2 16 64 100 each Same 1,300 shared, plus 1 GB SBD LUN Multi writer shared disk on vSphere
Proxy 4 4 16 60 1,000 squid cache each Cache sized to match server channel volume
Backup target 1 NFS export 4,000 mgradm needs 3x current install size free

Volume layout on the server host:

Path Content Backing
/var/lib/containers/storage/volumes/var-spacewalk Packages and repo metadata Pure LUN 1, xfs
/var/lib/containers/storage/volumes/var-pgsql Postgres data Pure LUN 2, xfs
/var/lib/containers/storage/volumes/var-pgsql-walbackup WAL archive for online DB backup Pure LUN 2 or 3
/var/lib/containers/storage/volumes/var-cache Salt file client cache OS disk
/var/lib/containers/storage/volumes/etc-rhn, etc-salt, etc-tomcat Config OS disk, in backup

Red Hat family channels are heavy. SUSE quotes 360 GB per Red Hat product against 50 GB per SUSE product (install guide). Two Liberty majors with clones is where most of the 1 TB goes. Grow the LUN on Pure, then xfs_growfs, no LVM needed, same convention as the AAP DB nodes.

Proxy count: 10,000 minions over 4 proxies is 2,500 each, above the 500 to 1,000 guidance. It works on a 10 GbE datacenter LAN where the guidance assumes WAN branches. Start at 4, watch squid hit ratio and Salt return latency during the first full patch cycle, add proxies to 8 if patch windows run long. Proxies are cheap, add before tuning the server.

Configuration and commands

vSphere settings for the MLM VM, both options:

  • vSphere HA on, VM restart priority High, host isolation response Power off and restart.
  • DRS automation Fully automated. Option B needs a VM anti affinity rule for the two nodes.
  • Reservation: 32 GB RAM and 8 vCPU reserved so HA admission control always has room.
  • Data disks as independent persistent, thick eager zeroed, PVSCSI controller, same as the AAP DB nodes.
  • No VM level snapshot older than the current upgrade window. Postgres on a snapshot chain gets slow.

Option A server deploy on SL Micro 6.1:

# storage, two devices: channels then database
mgr-storage-server /dev/sdb /dev/sdc

# install and deploy
transactional-update run zypper in -y mgradm mgrctl
reboot
mgradm install podman mlm.example.com \
  --tz UTC --email [email protected] --emailfrom [email protected]

# status
mgradm status
mgrctl exec -- systemctl status salt-master
mgrctl exec -- spacewalk-service status

# register to SCC and pull channels
mgrctl exec -- mgr-sync refresh
mgrctl exec -- mgr-sync add channel sll-9-updates-x86_64
mgrctl exec -- mgr-sync add channel sle-product-sles15-sp7-updates-x86_64

Backup and restore, the DR story for Option A:

# full backup to Pure NFS, nightly via systemd timer on the host
mgradm backup create /mnt/mlmbackup/$(date +%F)

# online DB backup with WAL archiving, enable once
mgradm backup db enable
mgradm backup db create

# restore on a rebuilt host
mgradm stop
mgradm backup restore /mnt/mlmbackup/2026-09-28
restorecon -rvF /var/lib/containers/storage/volumes/
mgradm start
mgrctl exec -- mgr-sync refresh

Restore is full only, no point in time. Backup needs 3x the current install size free (backup and restore, DB backup).

Proxy deploy:

# on the server, generate proxy config bundle
mgrctl exec -- spacecmd -u admin proxy_container_config_generate_cert \
  proxy01.example.com mlm.example.com 4096 [email protected] -o /tmp/proxy01.tar.gz

# on the proxy host (SL Micro 6.1)
mgr-storage-proxy /dev/sdb
transactional-update run zypper in -y mgrpxy
reboot
mgrpxy install podman /root/proxy01.tar.gz
mgrpxy status

Client bootstrap through a proxy, what AAP will call after a VM is created:

curl -sk https://proxy01.example.com/pub/bootstrap/bootstrap-liberty9.sh | bash
# or via API from AAP
# system.bootstrap(session, host, sshPort, sshUser, sshPassword, activationKey, proxyId, saltSSH=false)

Large deployment tuning on the server, only after 1,000 clients (tuning guide):

# inside the container
mgrctl term
vi /etc/apache2/server-tuning.conf        # MaxRequestWorkers 400, ServerLimit 400
vi /etc/tomcat/tomcat.conf                # Xmx 8G
vi /etc/rhn/rhn.conf                      # java.salt_batch_size=200, java.salt_presence_ping_timeout=4
vi /etc/salt/master.d/susemanager.conf    # worker_threads 24, timeout 60, gather_job_timeout 20
spacewalk-service restart

Option B Pacemaker skeleton, for cost and effort comparison only, not a validated procedure:

# both nodes, SLES 15 SP7 with SLE HA Extension registered
SUSEConnect -p sle-ha/15.7/x86_64 -r <HA_REG_CODE>
zypper in -y -t pattern ha_sles
crm cluster init -i eth0 -s /dev/disk/by-id/<sbd-lun> -y      # node 01
crm cluster join -c node01 -y                                   # node 02

# resources: vip, shared xfs, mgradm wrapper
crm configure primitive p_vip IPaddr2 params ip=10.0.0.50 cidr_netmask=24 op monitor interval=10s
crm configure primitive p_fs Filesystem params device=/dev/disk/by-id/<data-lun> \
  directory=/var/lib/containers/storage/volumes fstype=xfs op monitor interval=20s
crm configure primitive p_mlm systemd:mlm-wrapper op start timeout=600 op stop timeout=300 op monitor interval=60s
crm configure group g_mlm p_vip p_fs p_mlm
crm configure property stonith-enabled=true

mlm-wrapper would be a custom systemd unit calling mgradm start and mgradm stop. That unit is yours to maintain across upgrades. Ask SUSE if they will support it.

Pros and cons

Criterion A: single + vSphere HA B: Pacemaker active passive C: Hub
SUSE support posture Documented, tooling shipped Not documented for 5.x, custom wrapper Documented
RTO on host loss 3 to 6 min 30 to 90 s Per peripheral, same as A
RTO on data corruption 30 to 90 min restore Same, shared data Same
Protects against bad upgrade Snapshot revert No, shared volumes No
Extra SKUs None SLE HA Extension x2 MLM server subscription per peripheral
Extra VMs 0 1 idle full size 1 per peripheral
Ops skill needed podman, mgradm Pacemaker, SBD, shared disk on vSphere Hub sync, multiple consoles
Upgrade complexity mgradm upgrade Freeze cluster, upgrade, test failover Hub then each peripheral
Fits 10k one site Yes Yes Overbuilt
Fits 50k or multi site Needs Hub Needs Hub Yes

The honest summary: B buys about 4 minutes of RTO on the rarest failure mode and nothing on the common ones. A plus a rehearsed restore covers the failures that actually happen.

SKU and subscription impact

Approximate list of what changes. Prices are for SUSE to fill in. Do not sign on the HA line until they answer the support question.

Item Needed for Option A Option B Notes
SUSE Multi-Linux Manager Server The server 1 1 Covers SL Micro host and unlimited proxies, confirm proxy entitlement
MLM Lifecycle Management per managed node Every Linux client ~10,000 ~10,000 Ask about Unlimited tier at this count
SUSE Liberty Linux Former RHEL hosts, content and support ~8,000 ~8,000 Lite vs full support tiers, confirm which includes MLM node entitlement
SLES subscriptions Existing SLES hosts existing existing No change
SLE High Availability Extension Pacemaker on MLM nodes 0 2 Only value if SUSE supports the topology
SLES 15 SP7 base for HA nodes Option B host OS 0 included with MLM? SL Micro is included, SLES may not be, ask
AAP managed nodes Hosts AAP still automates ~2,000 Windows + ~500 Linux same Down from ~10,000, renegotiate tier

The AAP number is the budget story. Windows stays, Linux drops to the handful AAP touches for orchestration glue. Everything else is a Salt minion under an MLM node subscription, which is where the Satellite spend moves to.

Recommended path

Option A. Single tuned MLM 5.1 server on vSphere HA, four proxies, nightly mgradm backup to Pure NFS, online DB WAL backup, VM snapshot before upgrades, quarterly restore drill. This is what most enterprises run, because SUSE ships tooling for it and the failure modes that matter are covered by restore, not failover. Keep the Pacemaker option as a documented fallback if SUSE commits to supporting it in writing and the business demands sub minute RTO later.

Phased rollout:

  1. Now to week 2: confirm SKUs with SUSE, build production server VM from the vmdk image, attach Pure LUNs, register, sync Liberty and SLES channels. Keep the POC master for reference only.
  2. Week 2 to 4: deploy 4 proxies, build CLM projects with dev, test, prod stages, define system groups by environment, set maintenance calendars matching current AAP patch windows.
  3. Week 4 to 6: pilot 200 hosts, bootstrap through proxies, run one full patch cycle in MLM, run the restore drill once.
  4. Week 6 to 10: migrate remaining hosts in waves of 1,000 per week, apply large deployment tuning after 1,000. Each wave removes hosts from the AAP inventory.
  5. Week 10 to 12: cut AAP patch workflows over to MLM API calls, remove Linux hosts from AAP inventory, renegotiate AAP tier. Decommission Satellite.

Gate between phases: restore drill passed, patch cycle completed inside window, Salt return latency under 30 s at that wave size.

Interim path: standalone Salt multi master before MLM is licensed

MLM cannot run in production without a subscription, and Satellite still owns content. So the pivot is open source Salt for orchestration now, MLM for content and scheduling later. Nothing built in the interim is thrown away except the two interim master VMs.

flowchart LR
    subgraph Phase 0 now
        SAT[Satellite\ncontent only] --> MIN0[Minions]
        SM1[Salt master 01] --> MIN0
        SM2[Salt master 02] --> MIN0
        GIT[(git states, pillar, reactor)] --> SM1
        GIT --> SM2
        AAP0[AAP] -->|salt-api| SM1
    end
    subgraph Phase 1 at Satellite expiry
        MLM[MLM server + proxies] --> MIN1[Minions re pointed]
        GIT --> MLM
        AAP1[AAP] -->|XMLRPC| MLM
    end
    MIN0 -.re bootstrap in waves.-> MIN1
Loading

Interim master design, active active, no SLE HA SKU:

Item Value
Masters saltmaster01, saltmaster02, Liberty 9 or SLES 15 SP7, 8 vCPU, 32 GB, 100 GB
Salt version 3006 LTS, matches MLM 5.1
PKI Generate on 01, copy /etc/salt/pki/master/master.pem and master.pub to 02
Accepted keys /etc/salt/pki/master/minions synced 01 to 02 by rsync every minute, or shared Pure NFS export
Fileserver gitfs on both, same repo, base only
Pillar ext_pillar git on both
Job cache Local, or a Postgres returner on the existing AAP DB VIP if AAP needs job history
API salt-api with rest_cherrypy on both, F5 VIP saltapi.example.com for AAP
vSphere HA on, anti affinity rule between the two

Minion config for the interim, both masters listed, each minion talks to both:

# /etc/salt/minion.d/site.conf
master:
  - saltmaster01.example.com
  - saltmaster02.example.com
master_type: str
master_alive_interval: 60
recon_default: 1000
recon_max: 60000
recon_randomize: True

With master_type: str and a list, the minion connects to both and either master can run jobs. Do not schedule the same job from both. Use one master for scheduled work and the other as the API target, or run the scheduler only on 01.

Master config on both, identical, in git:

# /etc/salt/master.d/site.conf
interface: 0.0.0.0
worker_threads: 24
timeout: 60
gather_job_timeout: 20
auto_accept: False
fileserver_backend:
  - gitfs
gitfs_remotes:
  - https://git.example.com/infra/salt-states.git:
    - base: main
ext_pillar:
  - git:
    - main https://git.example.com/infra/salt-pillar.git
rest_cherrypy:
  port: 8000
  ssl_crt: /etc/pki/tls/certs/saltapi.crt
  ssl_key: /etc/pki/tls/private/saltapi.key
external_auth:
  pam:
    aap_svc:
      - .*
      - '@runner'
      - '@jobs'

Bootstrap of all Linux hosts, one AAP job, one time. AAP installs the minion, drops the config above, and pre seeds the key so no manual accept:

# on saltmaster01, pre generate a key per host
salt-key --gen-keys=vm1234.example.com --gen-keys-dir=/tmp/keys
cp /tmp/keys/vm1234.example.com.pub /etc/salt/pki/master/minions/vm1234.example.com
# rsync picks it up to 02 within a minute

# AAP playbook pushes to the host
# dnf install -y salt-minion (Satellite repo, EPEL or Salt Project repo)
# copy vm1234.example.com.pem and .pub to /etc/salt/pki/minion/minion.pem and minion.pub
# write /etc/salt/minion_id and minion.d/site.conf
# systemctl enable --now salt-minion

After bootstrap that host leaves AAP inventory. AAP reaches it through salt-api only:

curl -sk https://saltapi.example.com:8000/run \
  -H 'Content-Type: application/json' \
  -d '[{"client":"local","tgt":"G@env:prod","tgt_type":"compound","fun":"state.apply","arg":["patching"],"username":"aap_svc","password":"...","eauth":"pam"}]'

Rules that keep Phase 0 compatible with Phase 1:

  • Base environment only, no saltenv branching. MLM CLM does staging later.
  • Target on grains you set yourself (env, role, site), never on hostname globs. MLM system groups map onto the same grains.
  • No custom modules that depend on the job cache or on multi master behaviour.
  • Patching in Phase 0 is pkg.upgrade against Satellite repos. In Phase 1 it becomes an MLM patch action. Same state name, different backend.
  • Reactor only on failure events, AAP pulls everything else. Same rule as the MLM section.

Cutover to MLM per wave: register the host with the MLM bootstrap script through its proxy, which rewrites minion.d to point at the proxy, then delete its key from the interim masters. Roughly 15 minutes per 1,000 hosts of automation time, run it from AAP.

Before committing: verify current Salt Project package hosting and support status under Broadcom for 3006, and confirm the AAP contract counts managed nodes at renewal so the reduction lands in the budget.

Questions for SUSE

  • Is a Pacemaker active passive MLM 5.1 server a supported configuration? If yes, where is the procedure and which resource agent do you support for the containers?
  • If not supported, what is your written RTO and RPO guidance for a single server with mgradm backup?
  • Does the MLM server subscription include SLES 15 SP7 as the container host, or only SL Micro?
  • Proxy entitlement: unlimited proxies under one server subscription, or per proxy?
  • Lifecycle Management node pricing at 10,000: is there an Unlimited tier and where is the break even?
  • Liberty Linux Lite vs full: which tier includes the MLM node entitlement, and is Liberty 9 fully current with RHEL 9.7 errata?
  • Salt 3006 LTS is current in 5.1. What is the 3008 timeline and does it change minion requirements on Liberty 8?
  • Point in time DB recovery: release notes say manual workflow only. Is there a documented procedure?
  • Ansible control node integration in 5.1: does using it count MLM as an AAP execution environment for licensing on the Red Hat side?
  • Consultancy days: SUSE recommends consultancy above 10,000 clients. What does that engagement cost and cover?

Risks, open items, references

Risk Impact Mitigation
Liberty Linux errata lag vs RHEL Patch compliance gap on 8,000 hosts Ask SUSE for errata SLA, sample check 5 recent RHSA against Liberty channel
Restore never rehearsed Long outage on the one failure that matters Quarterly drill, restore to a scratch VM, time it
Proxy at 2,500 clients on LAN Slow patch windows Watch squid hit ratio, add proxies to 8
Postgres on VM snapshot chain Server slows after upgrade Delete snapshot within 24 h of a passed upgrade
Two inventories of record during migration AAP and MLM disagree on host state AAP reads MLM API for Linux, MLM is truth once a host is bootstrapped
SLE HA bought and unused Wasted SKU Do not buy until SUSE confirms support in writing

Open items:

  • Confirm Pure LUN sizes with storage team, 1,000 GB channels, 300 GB DB, 4,000 GB NFS backup
  • Decide whether existing AAP EDA calls MLM API or MLM Salt reactor calls AAP, one direction only
  • Map current AAP patch job templates to MLM system groups and maintenance windows
  • Activation keys per environment and OS, naming convention
  • Firewall: minions to proxies 4505, 4506, 443. Proxies to server 4505, 4506, 443, 5269. AAP to server 443

References:

⚠️ **GitHub.com Fallback** ⚠️