Wiki home

Documentation index · Source repository

Deploy the stack using the Swarm deployment guide. It runs the Controller, one Agent per selected node, Prometheus, and Grafana. The data source and KP2PLab Experiment Analysis dashboard are provisioned automatically. Use sh scripts/swarm.sh access to find the published control-node addresses; the examples below use control-node as a placeholder for that host.

The dashboard is written in English, and the Swarm stack sets GF_USERS_DEFAULT_LANGUAGE=en-US for the Grafana UI; user or organization preferences can override that UI default. See Grafana language preferences.

Grafana initializes its SQLite database on first startup, which may take several minutes depending on disk performance. Follow initialization with sh scripts/swarm.sh logs grafana. Subsequent starts reuse the existing database.

The local SQLite database uses WAL mode. Both the database and WAL files are retained in the same Grafana named volume. See the Grafana database settings.

The Swarm stack disables anonymous Grafana access and requires GRAFANA_ADMIN_PASSWORD. Its setup helper stores credentials in the manager's configuration; sh scripts/swarm.sh credentials prints the configured login. On startup, this stack synchronizes the original Grafana administrator password to the configured value while preserving the existing username and data.

Swarm publishes the Prometheus and Grafana ports on the control node. It also publishes each Agent's dedicated metrics listener on its own node at KPL_AGENT_METRICS_PORT (default 9091). Use scripts/swarm.sh configure to set PROMETHEUS_PORT, GRAFANA_PORT, or the Agent metrics port, then redeploy to apply the changes.

The Dashboard header links to Prometheus and Grafana in new tabs. It preserves the current Dashboard scheme and host and substitutes the configured published ports. Direct access and SSH tunnels work when those browser-facing ports match the configured values. If a proxy changes the scheme/path or local forwarding uses different ports, open the actual monitoring addresses separately.

For Dashboard controls use Dashboard; for run recovery use experiments; for source exports and backups use saved results.

1. Use Run experiment in the Dashboard to run examples/monitoring.yaml. This small experiment includes both envelope and raw publications. Set Runs beside Run to 1–100 for sequential iterations. Each gets a separate result; failure or Stop batch cancels the remainder. See execution and repetition.
2. Select Run (run_id), Agent, and Topic in Grafana. Selecting multiple runs aggregates their traffic and latency samples; network configuration and bandwidth time series identify each run in their legends. Session delivery panels apply only Run, and bandwidth/control panels apply Run and Agent without Topic.
3. After an experiment finishes, set the time range to its execution window to view the recorded series. The default refresh interval is 5 seconds.

Prometheus scrapes /metrics on the Controller and Agents every 5 seconds. The Controller returns the metrics URLs of registered Agents through HTTP service discovery; Prometheus then reaches each Agent's host-mode port directly instead of a service VIP. It does not scrape Peers directly or add exporters to individual Peer containers. The Controller aggregates the existing Peer telemetry stream, so telemetry loss under scope: all still affects Controller-derived Peer metrics. Monitoring services use a separate Docker network; Peers receive no additional networks or permissions.

kpl_network_configured_loss_ratio is the configured packet loss ratio, not an observed loss rate. Configured delay is also distinct from measured RTT. The graft/prune/remove_peer events help analyze PubSub mesh changes; they are neither a TCP connection graph nor a complete mesh snapshot.

For scheduled network changes, configured gauges follow the latest configuration reported as successfully applied by the Peer. Peers with networkPending: "true" are excluded until that report arrives. Status reporting, Agent heartbeat and Prometheus scraping introduce observation delay; scope: all can delay the report further even while the local schedule continues.

Scheduled Peers emit network_config_applied for the initial application and each applied change, with fields.network, reference, after, revision, initial, and lateByMs. The initial event confirms the startup configuration; later events are recorded after tc returns successfully. Lateness is measured against the selected schedule deadline and includes initialization time for the startup event. Skipped intermediate revisions indicate late entry or timer catch-up. Clock basis, offset and uncertainty fields follow the existing synchronized-clock event convention. network_config_failed records the failed revision, reference, deadline offset and error, and the Peer exits with an error rather than continuing under unconfirmed conditions. These events are saved in events.jsonl; whether a scenario subsequently fails also depends on its readiness, publication and error-handling phases. See scheduled network changes.

Control counters use only run_id, agent_id, direction, and control_type labels. The Topic filter does not apply because IWANT and IDONTWANT contain no topic and one RPC may contain several topics. send records outbound queue admission rather than remote receipt, recv precedes later router policy checks, and drop is a local queue/size rejection rather than netem loss. Read RPC, entry, message-ID-reference, and PRUNE peer-exchange counts as separate units. Exact per-RPC topic counts and the per-Agent totals in metrics.json remain in the result ZIP. See experiment metric definitions.

For current relationships, the Dashboard's interactive topology uses Peer status snapshots to display transport, Kademlia routing-table, and GossipSub mesh layers independently. The Controller also saves sampled protocol graphs in observations.jsonl; these merge topic edges within each protocol and do not retain every intermediate transition or individual score pair. Prometheus has no corresponding complete graph history.

Grafana session panels use only the Run filter and aggregate receiver-pair counts, not averages of percentages. Bounds describe observed cohorts; they do not establish an unseen population. Starting-cohort delivery and stable coverage remain available despite publication-time availability warnings or measurementIncomplete; they are N/A only when the selected runs contain no known starting pair. For each aggregation, known starting = stable + departed + continuity-unknown. The legacy-publications panel identifies historical data excluded from these results. New panels use kpl_window_* only; historical kpl_delivery_* and kpl_propagation_latency_seconds keep their old meanings and must not be combined.

Overall deliver counts include local delivery and cannot be divided by publication counts to obtain reachability. TCP retransmissions can turn packet loss into delay rather than message loss. See the metric definitions and formulas. Late batches can correct window gauges and histogram buckets; query them directly, not with rate/increase. Grafana latency shows cumulative whole-run quantiles, not quantiles restricted to the selected time range.

Cumulative counters are independent of the web interface's 300-event recent buffer. They reset when the Controller process restarts, and historical events.jsonl files are not replayed automatically. Time series already stored in Prometheus remain available, and rate/increase handle observed counter resets. They cannot recover events that disappeared before a scrape or telemetry that failed to arrive. Because increase estimates interval growth from scrape samples, it does not always exactly match integer cumulative event counts.

Deleting a saved Run releases its Controller event counters, metric initialization caches, detailed live accumulator, phase timing history and recent-event references. The durable deletion marker prevents late telemetry or heartbeat initialization from recreating these counters, including after restart. Terminal Peer inventory remains available for lifecycle fences. Prometheus retains previously scraped samples under its own retention policy. Completed Runs that remain saved retain their live accumulators until deletion or Controller restart.

KPL resource usage covers each Agent container and its owned running Peer containers. CPU is a percentage of the Docker host's total logical CPU capacity: 100 × occupied cores / host logical CPUs. For example, four busy cores on a 16-CPU host are 25%. The Agent reads online_cpus (or legacy per-CPU counters) from Docker stats, with a bounded, cached /info fallback. This denominator is host capacity, not the Agent container's CPU quota or configured Peer capacity. The numerator includes only KPL containers, not unrelated host processes. Memory is the sum of container cgroup usage; the default display is working set after subtracting inactive file cache, and raw usage including cache remains available.

Three scopes are available together: Agent + Peers (the existing total), Agent only (the Agent container), and Peers only (all owned running Peer containers). They reuse the same Docker reads, timestamp, and host CPU capacity; the split adds no polling or Docker requests. Each scope has its own measured/expected container counts and freshness checks. Agent-only container usage is distinct from the existing process_* CPU and resident-memory panels, which remain available.

Fleet CPU is capacity-weighted over Agents with fresh samples and known CPU counts: 100 × sum(occupied cores) / sum(host logical CPUs). It is not the sum or simple mean of percentages. For the standard one-Agent-per-host deployment, this represents KPL usage across the participating host capacity. Memory sums all available Agent measurements. If an older Agent does not report its host CPU count, memory remains available, CPU percentage is unavailable, and the fleet CPU coverage reports that omission. Existing raw core gauges and process counters remain available; Grafana's separate Agent-process CPU panel also uses host capacity as 100%, while process memory remains separate from combined KPL memory.

The Agent reads the local Docker Engine in a separate loop about every ten seconds, with four concurrent requests and an eight-second cycle deadline. It reuses CPU counters and one-shot stats reads; newly observed containers share a one-second warmup before their second read. It does not launch a Docker CLI process per sample or Peer, and metrics scrapes and heartbeats do not wait for Docker reads. The inventory at the start of the cycle defines the expected containers. Containers that start and stop entirely between samples and brief peaks cannot be reconstructed.

Partial samples remain usable: if 46 of 47 containers are measured, the displayed CPU and memory sum those 46 containers and show Partial · 46/47 containers. Missing usage is not extrapolated. An entirely unavailable sample, an offline Agent, or a sample older than 30 seconds contributes no live values. For a nonempty scope, zero successfully measured containers remains N/A. A successfully discovered empty Peer inventory is a complete zero-CPU/zero-memory sample; an unavailable inventory is never treated as empty. If only the Agent container fails measurement, available Peer measurements are still recorded, and vice versa. Scope totals use their own measured-Agent counts and CPU denominators, so fleet percentages with different missing samples need not add up to the combined percentage. Compare coverage before interpreting a resource drop during churn.

The Controller exposes agent_id gauges: kpl_agent_cpu_usage_percent (host percent), kpl_agent_cpu_capacity_cores (its denominator), kpl_agent_cpu_usage_cores (raw occupied cores), both kpl_agent_memory_working_set_bytes and kpl_agent_memory_usage_bytes, kpl_agent_resource_sample_timestamp_seconds, kpl_agent_resource_sample_complete, and kpl_agent_resource_containers (kind="expected" or "measured"). The complete flag stays zero for partial samples even though their values are exported. Measured-container counts are zero for stale/offline/unavailable samples. Grafana shows per-Agent and weighted fleet CPU, summed memory, and the percentage of expected containers measured. The existing five-second Controller scrape stores history without per-container TSDB labels.

The additional kpl_agent_component_* gauges mirror the same CPU, memory, timestamp, completeness, and coverage suffixes, with agent_id and component="agent" or component="peers" labels (coverage also has kind). Existing combined kpl_agent_* series keep their names and labels. Grafana adds Agent-container and Peer-container CPU/memory panels, component fleet totals, and expected/measured container counts. These are a fixed two components per Agent, not per-Peer series.

In Agent Status, Current CSV exports current values, complete/partial status, coverage, and the fleet aggregate. CPU uses cpu_percent_host; cpu_capacity_cores records the denominator. Select a period from five minutes to 24 hours for History CSV, or Summary CSV for counts, sample means, minima, maxima and first/last timestamps. History CPU is cpu_percent with unit percent_host, memory is in bytes, and coverage is exported as separate containers_expected and containers_measured series. Partial samples are included in means; missing usage is not estimated. Samples are not interpolated and averages are not exact time integrals. CPU percentages use a new gauge so old one-core percentages are never mixed in or divided by today's CPU count; historical normalized CPU is available only after upgraded Agents and Controller start reporting it. History remains in Prometheus, not experiment ZIPs, SSE or NAS run logs.

Current, history, summary, and interval CSVs include available rows for all three scope values: agent_and_peers, agent, and peers. Group by scope when analyzing downloads; adding all three would double-count the same usage. Component history begins after the Agent and Controller are upgraded and Prometheus scrapes the new gauges. Older Agents and historical combined-only recordings do not supply a breakdown and are not backfilled with zeros or estimated from process counters.

Interval measurement in Agent Status adds Start measurement and Stop measurement. Starting saves the Controller's current time; stopping fixes the end time. Select a completed interval for Interval CSV or Interval summary CSV. One active interval is shared across browsers, continues while the page is closed, and survives a Controller restart when its data directory is retained. An interval ends automatically at 24 hours. The latest 100 interval records are kept in <data-dir>/agent-resource-measurements.json, outside runs; starting the next interval removes the oldest record when this limit is reached. Existing current and recent-history exports remain available.

Use Delete interval beside the interval downloads to remove the selected completed measurement. Stop an active measurement first. Deletion persists across restarts and removes the saved interval/download entry; the shared Prometheus history and downloaded CSV files are retained.

In Agent Status → ⚙️ → Disable Agent, participation is saved per Agent ID. Disabled Agents show Excluded for hardware usage, contribute neither values nor denominator capacity to fleet totals, and disappear from topology and new placement candidates. The Agent and existing Peers keep running. Controller resource gauges omit them on the next scrape; process-only Agent targets disappear on the next discovery refresh. Past history remains available and is not retroactively filtered or erased; an interval crossing a disable contains samples collected while that Agent was eligible. Enable the Agent again to resume participation.

Interval boundaries select Prometheus scrape timestamps in (startedAt, endedAt]. They do not start another collector or change Agent sampling frequency. A CPU value summarizes the preceding Agent sampling period (normally about ten seconds), which can overlap the start boundary; repeated scrapes can contain the same latest Agent sample. Very short intervals may contain no samples. Closing or restarting the Controller can leave gaps, which are excluded from statistics. Stopping works without Prometheus; downloads can be retried after it recovers. The interval index stores only timestamps, not a sample archive: download availability depends on Prometheus retention and export size limits. Save the CSV for long-term use.

Swarm automatically uses the task name in KPL_SWARM_TASK_NAME to identify the Agent container and the existing Docker socket mount for read-only resource requests. Custom Agent launches can use --self-container <container-name> and --docker-socket <socket-path>; a missing identity/socket produces unavailable KPL totals rather than reporting Peers alone. Controller history exports use --prometheus-url / KPL_PROMETHEUS_URL: the stack sets http://prometheus:9090, while a standalone Controller defaults to http://127.0.0.1:9090. Current values work even when Prometheus is unavailable. History downloads require the same Dashboard login and are limited to two concurrent requests, 15 seconds, 16 MiB of upstream response, and 250,000 samples; choose a shorter period when the export is too large. Rebuild/redeploy Controller and Agents and apply the versioned Grafana dashboard config to enable collection; old recordings cannot acquire historical resource samples retroactively.

Controller telemetry admission is bounded before body decoding; its exact limits and retry contract are defined in API. The Controller exports kpl_controller_telemetry_inflight_requests (decoding or committing), kpl_controller_telemetry_waiting_requests (undecoded waiters), and kpl_controller_telemetry_rejected_total (requests rejected before decoding). Rejections do not mean that accepted telemetry was discarded; correlate them with Agent spool backlog and storage latency. /metrics allows two concurrent scrapes and returns 503 for excess requests. Its admission slot stays occupied until collection and response finish, even if the scraping client times out.

Live session-window-v1 metrics keep their session evidence without retaining a second legacy receiver/cohort index. Their cache retains aggregate latency histogram buckets instead of another copy of every propagation sample. Offline analysis still reconstructs detailed receiver groups and samples from saved logs. Out-of-order evidence and matured delivery deadlines update the same exact measurements. Metric reconstruction uses per-run locks, so slow aggregation does not hold the inventory or persistence lock while waiting. Dashboard SSE shares bounded snapshot work and releases canceled readers.

These changes do not cap the entire Controller heap: run measurement evidence remains available for late corrections until result deletion, and terminal Peer identities remain for lifecycle fencing. Memory can therefore grow with retained runs and observations. On the Controller host, compare MemAvailable and file/slab cache with the Controller's process RSS, Go heap, and container working set, and inspect Prometheus/Grafana separately. Low MemFree alone cannot establish a Go leak. No forced collection or kernel cache clearing is part of normal operation.

The Agent releases topology/score snapshots once a Peer container is confirmed removed, including failed Peer processes. Acknowledged terminal history is reduced to the most recent 64 records; older compact identities and evicted run fences stay in the Agent's local peer-history/ directory. It is not under runs. Retired Peers share one bbolt database file, peer-history/nodes, instead of one JSON file per Peer; runs/ and fences/ still contain per-run metadata. Lookup uses a disk-backed key, and Controller re-registration replays bounded pages without loading every historical record into the Go heap or holding a database transaction during a network send. Pending acknowledgements and failed container removal are never discarded. If completed records awaiting retirement or termination markers waiting for event-queue space reach 4,096, new admission pauses with 503 until delivery/storage recovers; existing Peers continue. Keep the Agent data volume to preserve retired-ID and generation checks.

At first startup, existing peer-history/nodes/ JSON records are copied in bounded batches, validated, and installed as the database using an atomic path exchange. This conversion requires a Linux filesystem with rename-exchange support; on failure, the original files remain and startup reports the error. The old directory becomes nodes.migrating/; maintenance compares records with the database and removes at most 128 matching files per pass. A restart resumes this cleanup. The converted nodes path deliberately blocks older Agents that expect a directory, preventing silent loss of duplicate-ID protection. Keep using an Agent that supports this format; draining telemetry alone does not make the converted history compatible with older images.

The Agent /metrics endpoint exposes kpl_local_history_records{kind="peers"|"run_fences"}, kpl_local_history_retired_total{kind="peers"|"run_fences"}, and kpl_local_history_cleanup_errors_total, all labeled by agent_id. In steady operation, acknowledged history tends toward 64 Peer records and 128 cached run fences; pending work can temporarily increase these counts. Check these alongside kpl_local_cleanup_pending, kpl_local_telemetry_queue_events, go_memstats_heap_alloc_bytes, and process_resident_memory_bytes. Heap allocation, resident memory, and container working set are different measures: freed objects may not immediately reduce RSS/working set. No forced GC or artificial zeroing is used to hide memory usage. See the synthetic churn regression.

Peer exit waits now use the Docker Engine API over the local socket instead of keeping a docker wait subprocess alive for every Peer. The wait connection pool is separate from resource sampling and has no lifetime deadline; cancellation still triggers the existing container-removal path. --docker-socket selects this socket as well as the resource sampler's socket; it must reach the daemon used by the Docker CLI for Peer creation. Config files for acknowledged, successfully removed Peers are deleted by history maintenance, including recent retained records; unrelated files, lifecycle history, and failed-cleanup records are preserved. This does not retroactively sweep orphan config directories left by older Agent processes.

Accepted telemetry payloads stay in the local disk spool. RAM retains an offset/run index plus a forwarding window of at most 5,000 events / 1 MiB of serialized events; one individually valid large event may exceed that window, up to the existing 10 MiB forwarding limit including the batch envelope. This is an encoded-data budget, not a Go heap limit. Recovery streams legacy spool arrays and loads only this window. Two requests may decode/admit telemetry concurrently. Up to 64 additional requests wait without decoding for at most one second to absorb short bursts. New batches receive 503 with Retry-After when the undecoded wait queue is full or expires or the pending backlog would exceed 50,000 events / 256 MiB. Pending disk and in-flight events both count. A full backlog also pauses new Peer admission, preventing continued churn from accumulating unlimited termination records; already-running Peers retain their final evidence, so those records can temporarily exceed the normal cap. Existing oversized backlogs are preserved and drained. Run completion waits for that run's disk backlog as well as active delivery; retries preserve event identities and order.

Queuing termination evidence and waiting for telemetry backlog checks happen outside the Agent state lock. A slow spool write therefore leaves heartbeat snapshots and other Peers' lifetime cancellations available. Creation admission checks cancellation and run fences again before reserving capacity; see admission and cleanup.

Small batches share an append-only .jsonl segment with a 4 MiB rotation target; a single larger accepted batch may exceed that target within the existing segment limit. Checksummed append-only ACK journals replace per-batch cursor-file replacement. One fully acknowledged active segment is retained for reuse, so a zero pending-event count does not mean an empty spool directory. This reduces inode/dentry churn and associated kernel-cache pressure. Forwarding copies persisted JSON directly into the byte-bounded batch without rebuilding Fields maps; recovery retains only the offset index. The 5,000-event cap lets small events use the existing 1 MiB budget instead of stopping at 250 events.

Pre-admission 503 responses now trigger safe Controller retry. Unpinned placement skips the busy Agent until Retry-After expires and can use another eligible Agent; pinned/single-agent placement waits for its selected Agent. The scenario context still bounds waiting, and stop/cancel releases reservations promptly. Arbitrary 503s, transport failures, and timeouts are not blindly re-placed because a Peer might already have been created. Admission waits are logged with Run and Agent IDs and can delay joins under sustained overload.

Old .json spool arrays remain readable. The format-v2.json marker prevents an older Agent from silently ignoring pending .jsonl evidence during rollback. Normal shutdown after delivery drains removes the empty active segment and marker. This clears only the spool guard: the converted Peer-history database still requires a compatible Agent image. Keep the Agent volume when updating or recovering.

Additional Agent metrics separate the sources of memory growth:

An initial rise after the first experiment can include runtime stacks, connection pools, and reusable buffers. Compare successive idle periods with similar active Peer counts and empty telemetry queues. A rising slab_reclaimable alongside cumulative retired Peers, with stable threads/live heap, points toward filesystem metadata rather than retained Go objects. The single-file history store removes the per-Peer inode growth; it does not cap database file size, OS page cache, or total container memory. See the observed history/cache trend.

Compare Agent only working set against process_resident_memory_bytes and Go live heap. A low process RSS with a high container reading calls for inspecting the cgroup file/kernel counters; forcing Go GC cannot free other processes' memory or explain all kernel charges. The memory regression checks queue recovery and delivery without relying on production GC forcing.

Agent Status → Controller & services shows the Controller, Prometheus, Grafana, and resource collectors above the Agent fleet. Each table row identifies the service and Docker host, container running state, host CPU percentage, working-set memory, and measured/expected container count. The Controller uptime is also displayed. Both Controller & services and Agents & Peers start their tables with a Total row. Service memory is summed; service CPU divides the summed measured CPU usage by the combined CPU capacity of the measured hosts, counting each host only once. Coverage indicates missing or partial measurements. Agent totals retain the combined, Agent-only, and Peer-only scopes and exclude disabled Agents; Peer counts and capacity include enabled, online Agents only. Running means Docker observed a running container; it does not assert application readiness. A stopped service, unavailable inventory, or stale collector report does not produce a zero CPU/memory measurement. Partial samples retain available containers, and reports become stale after 30 seconds.

The stack deploys one lightweight resource-monitor task on every eligible Linux Swarm node, independently of Agent placement labels and the UI's Agent enable/disable setting. Each collector uses the existing bounded Docker sampler: every ten seconds, at most four concurrent read-only Engine requests and an eight-second sampling deadline. New containers share one CPU-counter warmup. It selects only containers in its own stack with the container label io.kpl.resource-monitor=true, rechecks ownership, and excludes Agent and managed Peer containers to prevent double counting. Controller, Prometheus, Grafana, and the collector itself carry this label by default; the collector's own overhead is recorded separately as monitor. The configured control host also reports missing default services even if they never started. Other supporting services in the stack can opt in using the same container label. Service replicas on different hosts are distinct records; replicas on one host are summed using that host's CPU denominator.

The collector has local Docker socket access and cluster reporting credentials; the web Controller remains unprivileged and receives no Docker socket mount. A read-only socket mount does not restrict Docker API methods by itself: the collector implementation makes only GET requests. Collectors send bounded authenticated reports over the monitoring overlay. The Controller caches only the latest report per node; history remains in Prometheus, outside runs. The Dashboard fetches current service resources only while the Agent tab is visible, without adding service history to SSE. Lost hosts expire from the live inventory after ten minutes; their Prometheus history is retained.

The Controller /metrics endpoint adds kpl_service_cpu_usage_percent, kpl_service_cpu_usage_cores, kpl_service_cpu_capacity_cores, kpl_service_memory_working_set_bytes, kpl_service_memory_usage_bytes, kpl_service_resource_containers (kind=expected|measured), kpl_service_resource_sample_complete, and kpl_service_resource_sample_timestamp_seconds. Labels are service, node_id, and scope (controller, third_party, or monitor). kpl_service_running records 1/0 for a fresh known running/absent inventory; unknown or stale inventory omits it. kpl_service_resource_report_fresh distinguishes successful fresh reports from failures or stale reports. Service gauges remain independent of disabled Agents and Grafana's Agent filter. The Controller and supporting services Grafana section shows CPU, both memory measures, coverage, running state, and inventory freshness.

Start measurement / Stop measurement now includes these services in the same saved interval as all three Agent/Peer scopes. Current, recent-history, interval, and summary CSVs retain their existing columns and append entity_type, service, node_id, and node_name. Agent rows use entity_type=agent (current totals use agent_total); service rows use service and leave Agent identity columns empty. History uses stable node IDs and leaves node_name empty. Group by entity type, service, node and scope; do not merge two services because both have scope=third_party, sum percentages across hosts, or add the three overlapping Agent scopes. Running and report freshness are history metrics with unit boolean; their mean is the fraction of available scrape samples with value 1, not an exact uptime percentage. Gaps remain missing.

Rebuild/publish the KPL image and redeploy the stack to install the new collector, service container labels, Controller gauges, and versioned Grafana config together. Existing intervals remain usable, but service history begins only after the upgraded Controller is scraped. The collector requires Swarm containers and the labels described above; standalone/custom deployments without it show unavailable service samples. Inspect it with sh scripts/swarm.sh logs resource-monitor --tail 100.

The Controller automatically writes one JSON object per line under <data-dir>/logs. Local runs use data/logs; Swarm uses /var/lib/kpl/data/logs in the persistent controller-data volume.

Each record includes UTC timestamp, event, a server-generated requestId, remoteIp, HTTP method and path, response status, body bytes, durationMs, userAgent, authentication (anonymous, session, or internal), and user when known. Authentication records include outcome and a reason such as invalid_credentials, rate_limited, invalid_request, origin_denied, or login_required. A failed login records the submitted username when the JSON request is valid. Access and authentication entries share the request ID. SSE writes sse_open immediately when the response starts and sse_close when it ends.

Passwords, session cookies, Authorization headers, request/response bodies, Referer and URL query strings are excluded. Untrusted text is length-limited and JSON-escaped. remoteIp comes from the actual connection, without trusting client-supplied Forwarded or X-Forwarded-For; behind a reverse proxy this is the proxy's address. Successful authenticated Agent/Peer traffic and successful health/metrics/Prometheus-target checks are omitted to avoid per-event disk logging during experiments. Their errors and unauthorized attempts are still logged.

Each file rotates before exceeding 10 MiB, retaining the current file plus five backups (.1 newest through .5 oldest), up to approximately 120 MiB combined. Startup appends to existing logs. The directory uses mode 0700 and files 0600. Startup rejects an unusable log path; a later write error is reported in the Controller's ordinary service log without breaking HTTP requests, and subsequent writes retry. Records are appended without an in-memory queue; they are not individually fsynced. Include logs in Controller-volume backups when retaining access history. These files are not served by the Dashboard or included in experiment ZIPs.

For a local Controller, follow both files with:

tail -F data/logs/access.jsonl data/logs/auth.jsonl

For Swarm, use the manager helper to print the latest 100 lines as raw JSONL:

sh scripts/swarm.sh logs access
sh scripts/swarm.sh logs auth
sh scripts/swarm.sh logs access --tail 500
sh scripts/swarm.sh logs auth --tail all > auth.jsonl

--tail accepts a positive integer or all. These commands read the current file only; rotated .1 through .5 backups remain in the Controller volume. Output is a snapshot and can be piped to tools such as jq. Existing logs controller, logs agent, logs prometheus, and logs grafana continue to read timestamped service logs and also accept --tail.

The helper finds the running Controller task through the current manager Docker context, then reads its file with docker exec. By default the selected Docker daemon must host that task. If the Controller runs on another node (including a worker), create a Docker context for that node and pass it only for file reads:

# Replace user and control-node with your SSH login and Controller host.
docker context create kpl-control --docker "host=ssh://user@control-node"
sh scripts/swarm.sh logs access --context kpl-control
sh scripts/swarm.sh logs auth --context kpl-control --tail 500

Keep the caller's Docker context connected to a manager; --context on these two commands selects the daemon for the node check and file read, while task discovery stays on the manager. The helper verifies that the selected daemon is the task's actual node. It requires a running Controller with web logging enabled and reports missing files, unavailable tasks or a wrong node as errors. It does not start containers or change the stack. For continuous following or rotated backups, access the files directly on the Controller node.

A visible Dashboard uses /api/v1/stream?view=dashboard&topology=0 while Overview is Off (the page-load default), and topology=1 while On: an initial snapshot, then snapshot_delta updates or idle heartbeat events. Off avoids detailed graph construction and transmission while keeping population and experiment metrics live. Hidden pages disconnect and synchronize on return. The cumulative bytes of a long-lived stream are not a static page size. See Dashboard SSE for payloads, liveness timeouts, retry behavior, and legacy clients.

For excessive traffic or prolonged Reconnecting:

1. After an image update, reload the page and check that the browser Network panel uses the dashboard view above.
2. Compare sse_close.bytes with durationMs in the access log; inspect sse_open/sse_close and /api/v1/auth/session response status and duration together.
3. Check Controller/service load and reachability. A login error differs from a slow session probe; a browser disconnect does not establish an experiment failure. Consult saved run state before choosing recovery.

The earlier trace-heavy fixture and churn benchmarks are retained in performance evidence, with their validation limits.

Prometheus time series are stored in the prometheus-data named volume, and Grafana settings in grafana-data. Both survive ordinary container recreation. Prometheus retention is set to 15 days or 5 GB, whichever limit is reached first. The 5 GB setting is not a hard ceiling on disk usage: WAL, head data, and compaction require additional space. See the Prometheus storage documentation.

Swarm configs are immutable: Prometheus and dashboard configuration changes use versioned config references and require a stack redeployment. The provisioned originals are managed as files. To save a separate dashboard, sign in as an administrator and work with a copy. See the Grafana provisioning documentation.

# Check scrape target status.
curl http://control-node:9090/api/v1/targets

# Check service status and logs from the manager.
sh scripts/swarm.sh status
sh scripts/swarm.sh logs prometheus
sh scripts/swarm.sh logs grafana

The Controller target also uses a browser-reachable address: GET /api/v1/prometheus/controller-targets returns the control node's Swarm address and published KPL_HTTP_PORT. The deployment helper derives KPL_CONTROLLER_METRICS_URL and KPL_PROMETHEUS_EXTERNAL_URL on each deployment, including custom ports and IPv6. Prometheus uses the latter as --web.external-url for its own links. Internal DNS remains in discovery requests and the Grafana datasource. Prometheus containers must be able to reach the control node's published HTTP port. An unset Controller metrics URL returns an empty target list.

The Swarm stack discovers registered targets from GET /api/v1/prometheus/agent-targets. Use sh scripts/swarm.sh access to inspect the advertised URLs, ensure the configured port is free on every selected Agent node, and permit TCP traffic from the control node. If operators open an Agent metrics link directly, permit their browser's trusted management network as well; block untrusted sources. up{job="kpl-agent"} distinguishes successful scrapes from registered targets that are unreachable through a firewall or an incorrect Swarm NodeAddr.

Image versions are pinned to Prometheus v3.13.2 and Grafana 13.2.1. When upgrading, consult the official Prometheus downloads and Grafana Docker installation guide, then revalidate the configuration and dashboards.

Validate a new run on Linux and retain its ZIP, code revision or image digest, and Prometheus time range. The repository does not include the original ZIPs behind earlier monitoring run examples, so those historical counts are not an auditable baseline for the current session-window implementation.

For examples/monitoring.yaml, check:

Compare metrics.json with its exact events.jsonl prefix and export.json.exportedAt. Counter comparisons also require the same Controller lifetime and collection boundary. Heartbeats initialize zero baselines for add_peer and remove_peer, but an event before the first scrape can still be absent from an increase estimate. Delivery, duplicate, mesh-event, and latency totals depend on execution and observation; they are not fixed acceptance counts.

The metric implementation and regression cases are linked from experiment metrics. Use the development guide for Linux validation commands.

This section moved to dashboard.

See saved-result browsing.

See Dashboard panels.

This section moved to results.

This section moved to results.

Korean translation: 한국어.