Freshservice sync - SS67/project-docs GitHub Wiki

VSPHERE_PENDING.md

Every open item on the vSphere Freshservice sync workstream, with paths, scripts, playbooks, env vars, and nuances. As of 2026-09-30. Belongs in the vSphere project files, not the Intune set. Companion to the main context doc and the AAP environment reference.

Warning on numbers: all metrics below date from early August, before the pause. Cat 5 has drifted since. Re measure before acting on any count.

Last known state

  • vSphere: ~8570 VMs across 2 vCenters. Fresh: ~11880 assets, ~11621 unique UUIDs.
  • Cat 5: 3323 against baseline 3308 (+0.45 pct at the time, under the 20 pct halt).
  • fs_retire validated: first live run retired 100 at fs_retire_max_per_run=100, 100 pct accuracy on spot check.
  • Retire drain paused since early August with ~1169 eligible remaining.
  • Branch feature/sync-last-seen-tracking unmerged, stable for weeks.
  • Schedules running: snapshot every 15 min (xx:00/15/30/45), reaper hourly xx:20, inventory source update xx:25, sync_all_vms xx:35 every 1 to 2 hours. FS Apply and FS Retire manual.

Update 2026-09-30: scoped drain findings and decisions

Reported from the scoped work chat. Supersedes the numbers above where they conflict.

State changes:

  • Global Cat 5 now 5661 against baseline 3308, 71 pct growth, unexplained. All retires blocked, scoped and global, until a diagnostic explains it.
  • Drain is being attempted scoped first: two cluster groups (cmsp-ucs plus one more), about 350 VMs, 81 scoped Cat 5 candidates. Item label GAR-1.
  • Claim pending verification: sync_all_vms scoped to two clusters since mid July, leaving sync_last_seen stale fleet wide outside those clusters. Must be proven or disproven from the job template limit or inventory source config. If true, full fleet sync must be restored and at least two refresh cycles (12h plus) must pass before any global retire, because stale last_seen collapses defense in depth to a single layer (the reaper full pull).

Decisions (2026-09-30, recorded by architect chat):

  1. Scoped retire approved as Option A: scope filter on the Fresh side only in fs_retire.py or reaper output consumption, separate FS_REPORT_DIR so the scoped run gets its own baseline, first run captures baseline and retires nothing. Before relying on it, verify FS_REPORT_DIR is genuinely honored by fs_retire.py at runtime, if the env var does not exist yet, adding it is part of the change. Dry run first, cap 25 on the first live scoped run.

  2. Standing rule, permanent: the reaper's vSphere inventory pull is never narrowed, to any scope, for any reason. A partial pull turns live VMs into Cat 5, and stale sync_last_seen would then make them eligible. The full pull is a load bearing safety layer.

  3. The 81 scoped candidates must be checked against the full inventory (8809 hosts, inventory id 9) for cluster moves before any write. Any candidate matching a live VM elsewhere is excluded and logged.

  4. Option B (accept a new global baseline) rejected until the Cat 5 growth is explained.

  5. Global drain resume protocol above is amended: additionally blocked on the Cat 5 diagnostic, and on full sync restoration plus re stamp if the mid July scoping is confirmed.

  6. Sync freshness precondition for retire (2026-09-30 pm, user decision): retire runs always follow a recent full estate sync, mirroring the AAP inventory refresh pattern. Implementation in two parts. Part one, marker: sync_all_vms writes a completion marker /var/lib/awx/fs_reports/sync_last_full_run.json with finished_at, hosts_seen, per vCenter counts. Part two, guard: fs_retire refuses with exit 2 when the marker is older than FS_SYNC_FRESHNESS_HOURS (default 8, covers 1 to 2h schedule plus 3h30m runtime) or hosts_seen is below FS_SYNC_MIN_HOSTS (default 90 pct of expected estate). The host count check is the important half: a scoped or partial sync must not satisfy the guard, that is exactly the mid July failure mode. Optional convenience: an AAP workflow job template chaining sync_all_vms, snapshot refresh, reaper, retire for one button drain runs. With healthy schedules the guard alone suffices, chaining a 3h30m sync before every retire is not required.

Diagnostic to run first, read only, no approval needed:

  • Per vCenter host counts across the last 10 reaper runs. A partial or failed vCenter pull inflates Cat 5 in one step.
  • Histogram of the roughly 2300 new Cat 5 entries by fs_updated_at month and name prefix. The bulk import signature (dev1, chia, buil, pyte) means the external dumper is still active, which also escalates the P2 needs_review policy call.
  • sync_last_seen distribution of the delta entries.

Agreed sequence, 2026-09-30 afternoon

User decision: full fleet sync will be restored. Ordered runbook, each step verified before the next:

  1. Remove the sync limit (job template limit or inventory source scoping from mid July). Confirm full estate pull, ~8600 hosts, note per vCenter counts.
  2. Merge feature/sync-last-seen-tracking to main, flip project 38, force sync, smoke test sync_single_vm. Merge before the estate sync so it runs on main with the refresh rule.
  3. Full fleet sync run. Expect ~8600 refresh_stale PUTs (stamps ten weeks old), ~70 plus min API time inside the usual ~3h30m. Shares rate budget with snapshot job, do not stack a retire run on it.
  4. Verify stamps, do not assume: sync_last_seen age histogram over fs_vmware_assets.json, live VMs expected under 6h. Proof the second lock is back.
  5. Explain Cat 5 5661 with two or three reaper cycles on confirmed full pulls. Buckets: no stamp plus name prefix signature = bulk import (needs_review pile, escalates policy call). Frozen stamp, genuinely absent = real decom churn, legitimately eligible. Oscillating membership = partial pulls, stop and fix infra. Note: full sync never stamps Cat 5 assets (no VM to find), the stamp differential is the diagnostic signal.
  6. Spot check 5 to 10 eligible via pyvmomi (existence plus config.template). Clean plus explained growth = new baseline acceptable.
  7. GAR-1: cross check 81 vs full inventory for cluster moves, Option A build, dry run, live cap 25.
  8. Global drain capped 200 to 300 daily, drop cap when empty. Then needs_review policy, 1.5b, polish.

Steps 1 to 4 are one day including runtime. Step 5 one day. Draining early the following week.

Process note: an agent flagging an architecture change and stopping is correct behavior, not a stall. The loop is: agent flags, architect chat decides, decision lands in this file, scoped chat resumes with the decision quoted. Read only diagnostics are pre approved and never require a stop.

P1: Retire drain (close before Intune build, week of Sep 29)

Goal: drain ~1169 eligible Cat 5 assets, then drop the cap.

Resume protocol, in order, because weeks of drift change the ground:

  1. Check current Cat 5 count in /var/lib/awx/fs_reports/reaper_latest.json against baseline 3308 in /var/lib/awx/fs_reports/retire_baseline.json. If drift exceeds 20 pct the spike guard (FS_CASCADE_SPIKE_PCT=20) will halt with exit 2. That is the guard working, investigate the growth before overriding, remember external bulk imports have inflated Fresh before (the 1835 incident, name prefixes dev1, chia, buil, pyte).
  2. Re spot check 5 random eligible entries against vCenter directly, last validation is nearly two months old. Method used before: pyvmomi loop through both vCenters checking existence and config.template.
  3. Resume daily capped runs: job template FS Retire on instance group cp01_only, extra vars fs_apply: true and fs_retire_max_per_run: 200 (raise toward 300 as confidence returns). Roughly 4 to 6 runs drains it.
  4. Each run: eligibility is Cat 5 with sync_last_seen older than FS_DECOM_STALE_HOURS=12, report freshness must be under FS_REPORT_FRESHNESS_SECONDS=1800, so run within 30 min of a reaper run (reaper fires xx:20). Buckets: eligible, needs_review, too_fresh, already_retired.
  5. Verify per run: applied_fs_retire_TIMESTAMP.json in fs_reports, ledger lines in audit_ledger.jsonl, baseline only advances when errors are zero.
  6. After drain completes: remove the cap (fs_retire_max_per_run back to 0), record completion in the context doc.

Runtime note: Fresh API pacing ~0.5s per request, 200 retires is roughly 2 to 3 minutes of PUTs.

Relevant files:

tools/fs_retire.py                                        decom flow, cascade halt, cap
playbooks/fs_retire_run.yml                               wrapper playbook
/var/lib/awx/fs_reports/reaper_latest.json                symlink to newest reaper report
/var/lib/awx/fs_reports/retire_baseline.json              Cat 5 baseline for spike detect
/var/lib/awx/fs_reports/applied_fs_retire_TIMESTAMP.json  per run output
/var/lib/awx/fs_reports/audit_ledger.jsonl                append only ledger

Env vars: FS_APPLY (default false), FS_DECOM_STALE_HOURS=12, FS_CASCADE_SPIKE_PCT=20, FS_RETIRE_MAX_PER_RUN (0 = unlimited), FS_REPORT_FRESHNESS_SECONDS=1800.

P1: Merge feature/sync-last-seen-tracking to main

Goal: main carries the refresh rule and last seen tracking. Blocks Intune collector build, same repo, do not stack new work on an unmerged branch.

  1. Merge to main in git, no functional changes expected, branch stable for weeks.
  2. Flip the AAP project (id 38, path /var/lib/awx/projects/_38__fresh_vcenter_asset_sync on cp01) to track main, force a project sync.
  3. Smoke test: one sync_single_vm run, confirm reason codes delta, refresh_stale, unchanged still surface, confirm sync_last_seen stamps.
  4. Nuance: AAP UI inventory can be cached, if CLI dump and UI disagree after the merge, force inventory source resync before drawing conclusions.

What the branch carries, for the merge message: refresh rule in roles/freshservice_vm_sync/tasks/upsert.yml (forces PUT when sync_last_seen missing or older than fs_refresh_threshold_hours, default 6), fs_delta_ignore_fields: ["sync_last_seen_55000745586"] in defaults (prevents timestamp self triggering ~8000 PUTs per run), reason codes in output. The refresh rule is what makes fs_retire safe, without it live VMs age into eligibility.

P2: needs_review policy decision (2039 assets)

Not build work, a policy decision, stakeholder call. These are Fresh assets under our asset type with no sync_last_seen because our sync never touched them, dumped by an external CI/CD source (the bulk import finding). They sit in the needs_review bucket of every fs_retire run.

Three real options on the table:

  1. Leave forever, accept the noise in Cat 5 accounting.
  2. Manual classification in Fresh by asset owners.
  3. Slow burn auto retire: stamp sync_last_seen on first observation in Cat 5, age out at 30 or 90 days. Small code change in fs_retire.py, self resolving, recommended option to bring to the call.

Related open policy: whatever external system dumps these may still be dumping. Decision should include finding and gating that source, or the bucket refills.

P3: Phase 1.5b, fs_apply auto apply

Switch fs_apply from manual to scheduled for the safe categories only: Cat 1 single name match (stamp UUID onto existing asset) and Cat 2 unique name (rewrite UUID). Risky cases stay manual forever: multi matches, duplicate UUIDs, uuid_already_in_fs guard hits, bulk anomalies.

Mostly job template work: schedule FS Apply (a slot after reaper xx:20, before snapshot xx:30, so freshness holds), extra var fs_apply: true. Guards already in the code carry the safety: report freshness 1800s, drift guard vs current snapshot, cascade halt at FS_CASCADE_HALT_PERCENT=5 or FS_CASCADE_ORPHAN_CEILING=500, exit 2 refuses everything.

Files: tools/fs_apply.py, playbooks/fs_apply_run.yml, output applied_fs_apply_TIMESTAMP.json plus ledger.

P4: Phase 2, EDA driven single VM sync

New project of similar scope to Phase 1, replaces hourly polling with event driven sync at scale. Not started.

  • Events: VmCreatedEvent, VmRemovedEvent, VmRenamedEvent, VmReconfiguredEvent.
  • Needs: EDA rulebook on eda01/eda02, decision tree mapping events to playbooks (created → sync_single_vm seed path, removed → guarded retire path, renamed and reconfigured → upsert).
  • Design note from Sep discussions: same event driven pattern later aligns with Cisco ISE pxGrid for network devices, design the rulebook shape with a second source in mind.

P5: Operational polish backlog (all small, none blocking)

  1. Per VM jitter on refresh threshold: uuid hash mod 120 min added to fs_refresh_threshold_hours in upsert.yml, spreads the 6 hourly PUT bursts.
  2. Log lines on the two silent template skips in inventory/vsphere_inventory.py, line 615 (config.template is True) and line 624 (vm_is_template post check). Both currently skip with no log, burned an hour on wow-webapp once. Two one line additions.
  3. Batched helper: tools/fs_snapshot_lookup.py subprocess parses the ~30 MB snapshot JSON once per VM per run, full sync_all_vms takes ~3h30m and will not survive 10k+ VMs. Fix: single JSON load via set_fact run_once with trimmed payload. Biggest performance item on the list.
  4. vCLS skip pattern in the inventory script, matching the Jenkins inventory convention.
  5. Report retention on /var/lib/awx/fs_reports: grows forever, hourly reaper JSON plus 5 CSVs each since July, and cp01 disk now also carries fs_staging. Agreed approach: delete reaper and applied files older than 90 days, keep audit_ledger.jsonl forever. One cron or systemd timer on cp01.

P6: Items added during September (bookkeeping and observability)

  1. Context doc update: main context doc still says drain in progress at old baseline. Paste the agreed correction (drain paused since early August, resume protocol, merge pending) into its Current state section.
  2. Build vs plugin decision record: half page documenting why the Freshservice vCenter Cloud Discovery app was rejected, so it is never relitigated. Points: supported vCenter 6.5/6.7 only with 8.x unconfirmed, discovery not reconciliation (no reaper/apply/retire equivalent, could not repair the dirty 11880), no control of match key or custom fields (instance_uuid, sync_last_seen), blind to vSphere tags and custom attributes (separate SOAP field manager and vAPI REST stacks the app never calls), Freshservice deprecated its cloud compute asset hierarchy May 2025 and plugin riders ate that migration, Device42 powered advanced discovery is the real buy alternative and still lacks the patch compute and ledger semantics.
  3. Grafana observability, proposed not committed: Alloy or Promtail on cp01 tailing audit_ledger.jsonl into Loki, exporter parsing reaper_latest.json into Prometheus metrics (Cat 1 to 5 counts, healthy, eligible, needs_review), alerts on exit 2 halts, Cat 5 spikes, ledger silence in scheduled windows. Open question never answered: does a Loki and Prometheus stack already exist or is this net new infra. Ledger schema note if pursued: consistent keys per line, timestamp, action, source, asset id, field, old, new, reason, run id, result.

Reference: guards and env vars across the tools

fs_retire.py   FS_APPLY, FS_DECOM_STALE_HOURS=12, FS_CASCADE_SPIKE_PCT=20,
               FS_RETIRE_MAX_PER_RUN=0, FS_REPORT_FRESHNESS_SECONDS=1800
fs_apply.py    FS_APPLY, FS_CASCADE_HALT_PERCENT=5, FS_CASCADE_ORPHAN_CEILING=500,
               FS_REPORT_FRESHNESS_SECONDS=1800
fs_reaper.py   inherits VMWARE_*, FS_*, FS_REAPER_DEBUG
snapshot       FS_API_KEY, FS_DOMAIN, writes /var/lib/awx/fs_snapshot/fs_vmware_assets.json
lookup helper  FS_LAST_SEEN_KEY=sync_last_seen_55000745586, returns last_seen_age_seconds (-1 missing)
inventory      VMWARE_HOST (comma list from Vault secret/infra/dev/vcenter), VMWARE_USER,
               VMWARE_PASSWORD, VSPHERE_INV_NO_CACHE, VSPHERE_INV_DEBUG

Standing nuances that bite these items: jinja2_native=True required in ansible.cfg, both vault bound AAP credentials need Path to Auth explicitly approle, all pinned templates run on cp01_only because state lives on cp01 local disk, Fresh rate limit is per API key so retire runs and snapshot refresh share the budget.

Suggested order

  1. This week: drain resume (passive, daily babysit) + merge (one hour) + context doc paste.
  2. Opportunistic: retention timer, the two log lines. Minutes each.
  3. Stakeholder call when schedulable: needs_review policy, bring the slow burn option.
  4. After Intune MVP ships: Phase 1.5b (a config afternoon), then batched helper before fleet growth, then Phase 2 EDA as its own project with its own file set.