Controller Performance Investigation and Benchmarks

Wiki home

Documentation index · Source repository

This is implementation evidence, not a capacity guarantee. For operating an existing deployment, start with Agent capacity and monitoring. The investigation below records earlier measurements; it has not been rerun merely by reorganizing this documentation.

The 2026-09-22 investigation observed 65,227 retained Peer records. Recent five-minute Prometheus averages showed about 7.17 CPU cores and approximately 526 MB/s of Controller memory allocation. Allocation rate is distinct from live memory size; it also includes the cost of creating and reclaiming short-lived objects.

Investigated paths and implementation changes

• /api/v1/bootstrap previously generated a full snapshot to obtain addresses, copying/sorting all historical Peers and calculating metrics/topology. It now reads ready boot addresses for that run from the active-node index.
• Discovery and occupancy checks for partial heartbeats also use the active-node index. Partial heartbeat processing no longer scans the entire history for omitted nodes.
• Dashboard SSE excludes terminal nodes before snapshot construction, avoiding copies and sorting that would later be discarded. Full inventories and saved results retain history.
• Agent, experiment, and recent-event REST endpoints copy only the requested lists. Agent registration builds Agent data without a full Peer snapshot.
• Prometheus reuses propagation-latency histograms for unchanged completed experiments. New events, delayed evidence, and delivery-window expiry still update summaries and histograms. Metric definitions and saved-result calculations remain in effect.
• Agent/Controller event-size validation inspects and discards each encoded event's length instead of allocating the complete batch again. JSON escaping, the exact 10 MiB boundary, and rejection of a batch containing an invalid later event are retained.

Reproducible benchmarks

Recorded on Intel Xeon E5-2620 v3 with GOMAXPROCS=2 and identical inputs. Lookup fixtures contain 65,000 terminal Peers and 100 active Peers. Event validation uses 250 events with 4 KiB strings. MB and KB mean 1,000,000 and 1,000 bytes.

Path	Before: time/op	After: time/op	Before: allocation/op	After: allocation/op
Bootstrap address lookup	196.56 ms	0.111 ms	149.22 MB	28.86 KB
Dashboard frame creation	242.60 ms	1.132 ms	171.83 MB	320.91 KB
Event-size validation	4.33 ms	1.70 ms	7.05 MB	12.26 KB

After the change, active-node lookup allocated similar amounts with no terminal history and with 65,000 records. Production CPU also depends on active population, topology, event rates, and Docker load. These are local path benchmarks; overall CPU and arrival rates require another experiment after deployment.

GOMAXPROCS=2 go test ./internal/controller ./internal/model \
  -run '^$' \
  -bench 'Benchmark(BootstrapChurnHistory|DashboardChurnHistory|ValidateTelemetryBatch)$' \
  -benchtime=300ms -benchmem

Applying these changes requires rebuilding and redeploying Controller and Agent images; deleting history is not a prerequisite. Compare the following metrics and Controller churn join schedule delayed logs using the same scenario and active Peer population:

rate(process_cpu_seconds_total{job="kpl-controller"}[5m])
rate(go_memstats_alloc_bytes_total{job="kpl-controller"}[5m])
sum by (state) (kpl_nodes)
kpl_local_telemetry_queue_events

SSE payload evidence

An earlier synthetic trace-heavy 20-second fixture (21 updates) reduced JSON payload from 85,767,024 to 170,144 bytes when using the compact Dashboard stream. This is a regression-fixture measurement, not a mobile-network measurement or a bandwidth guarantee for arbitrary cluster sizes. Dashboard SSE owns the current format; connection diagnosis describes operational checks.

---

Korean translation: 한국어.

Agent memory retention regression

The 2026-09-28 code review found that Agent process records and run-fence maps grew with historical churn even after successful Controller acknowledgement. Cleaned failed processes also retained their last topology/score collections. The Agent now moves older acknowledged terminal records and evicted fences to local storage, releases removed-container topology, rebuilds oversized maps after draining a backlog, and retains only compact offsets for unacknowledged telemetry on disk. Live heartbeats stay independent of history writes; Controller restart history replay uses bounded pages.

A synthetic regression inserts 256 terminal Peers per round, acknowledges them, and runs retirement for 12 rounds (3,072 Peers total). Retained terminal history stays at 64 after every round. An earlier run of this fixture, before single-file history storage, reported approximately 1.23 MiB after the first round and 1.06 MiB after the final round following explicit test-only GC. Those historical values include the test process; they are neither measurements of the current storage implementation nor production memory limits. Additional tests cover failed delivery, failed removal, local storage failure and recovery, duplicate IDs after retirement/restart, stored generation fences, crossed admission/retirement, and bounded replay after Controller restart. Real Docker/container memory under a sustained deployed workload still requires observation after rollout.

go test ./internal/agent -run '^TestChurnHistoryMemoryIsBoundedAndRetiredIDsRemainFenced$' -count=1 -v
go test -race ./internal/agent ./internal/controller -run 'History|Retire|RunFenceCache|Heartbeat' -count=1

The disk-backlog regression enqueues 1,024 unique 32 KiB payloads (32 MiB total), reopens the spool, checks its bounded window, and drains every event in original order. On a local Go 1.27.1 run, test-only post-GC heap was 1.21 MiB before enqueue, 1.25 MiB with the full backlog on disk, and 2.52 MiB after recovery with 31 loaded events. These are synthetic test-process measurements, not deployed container RSS or a promised production memory ceiling. Additional cases cover legacy spool files, partial acknowledgment failures, byte backpressure, busy decoders, and 64 concurrent cancelable Engine waits that start no child processes.

go test ./internal/agent -run 'TestTelemetryDisk|TestTelemetryRecovery|TestTelemetryPartial|TestTelemetryDecoder|TestTelemetryByte|TestDockerEngine' -count=1 -v

Telemetry throughput and filesystem churn

A further regression exercises 256 small append/ack cycles and verifies that the same payload and ACK inodes are reused. Tests also cover partial payload/ACK writes, mixed legacy recovery, raw JSON identity/order, an empty-but-reusable active segment, and rollback-format protection. This addresses filesystem churn that a post-GC Go heap test does not measure.

A local comparison on an Intel Xeon E5-2620 v3, using the same 2,000-event window and byte budget, measured about 63.3 ms / 10.24 MB allocated for decoding and re-encoding versus 7.33 ms / 0.61 MB allocated for copying persisted JSON. These are per-operation benchmark allocations and latency, not retained heap or production container memory. Recheck long-running cgroup slab_reclaimable, anonymous, process RSS, and live Go heap after rollout. No host-wide cache dropping or periodic forced GC is used.

go test ./internal/agent -run 'TestTelemetryLog|TestTelemetryRaw|TestTelemetryFormat' -count=1
go test ./internal/agent -run '^$' -bench '^BenchmarkTelemetryForwardingWindow$' -benchmem
go test ./internal/controller -run 'TestDeferredAgent|TestAmbiguousAgent|TestAdmissionRetry' -count=1

Retired Peer history and kernel cache

Read-only Prometheus samples on 2026-09-29 showed continued container-memory growth after the telemetry fixes. On one Agent, cumulative retired Peers rose from 401 to 3,133 while cgroup slab_reclaimable rose from about 1.1 to 7.5 MiB. OS threads stayed at 111 across that interval; live Go heap fluctuated around 3.2–7.8 MiB. The earlier first-workload increase also involved runtime threads/stacks. These observations identify a remaining filesystem-metadata trend, rather than proving that all container-memory growth has one cause.

The previous history format left one permanent JSON file per retired Peer, including an inode/dentry and negative lookup paths for new IDs. The Agent now stores retired records in a single indexed bbolt file. Recent in-memory records, per-run indexes, generation fences, delivery acknowledgements, and duplicate-ID protection remain. Existing JSON histories migrate before startup reconciliation; the installed records are verified before bounded cleanup of the old files. See migration and format compatibility.

A regression stores 512 retired Peers per run for 12 runs (6,144 records) and verifies that Peer history still uses one file, with separate per-run indexes. Migration/restart tests replay all 300 legacy records in three pages, retain corrupt or mismatched source evidence, and check whole-batch rollback, copied replay cursors, byte bounds, and cancellation. These tests verify storage behavior, not a deployed memory reduction. After rebuilding and redeploying Agents, compare idle-period slab_reclaimable, live heap, thread count, and Agent-only working set under the same workload. A return to the initial 20 MiB is not guaranteed; database pages and reusable runtime resources remain reclaimable or reusable according to the OS/runtime.

go test ./internal/agent -run '^TestHistoryStore' -count=1 -v
GORACE=atexit_sleep_ms=0 go test -race ./internal/agent -run 'Test(History|RunFence|Retir|Registration|Heartbeat|AgentHistory|BlockedConfig|ConfigWrite)' -count=1