Wiki home

Documentation index · Source repository

The dashboard and operational APIs require a login session. See authentication for public monitoring exceptions and internal service authentication.

The Controller persists web access and authentication attempts in <data-dir>/logs/access.jsonl and auth.jsonl. Passwords, cookies, authorization headers and query strings are excluded. See web access and authentication logs for fields, rotation and retrieval.

The runId query parameter on /api/v1/bootstrap is required. The registry returns only ready boot nodes with usable identity and address data from that run, so concurrent experiments cannot discover one another's bootstrap peers. /api/v1/discovery requires all three shown query parameters, excludes the requester, and returns configured topic participants rather than observed delivery or mesh outcomes. /api/v1/prometheus/agent-targets is a read-only operational endpoint used by the supplied Swarm Prometheus configuration; it omits offline Agents and Agents without a valid metrics URL.

The bootstrap response is an array of {nodeId, peerId, addresses} records (or null when empty). Unlike discovery, bootstrap does not filter on Agent online status. Discovery returns an array, empty when no candidates match, with an additional subscribed flag. It returns the full eligible candidate set; the requesting Peer applies rendezvous ranking, connection budgets, and retries as described in topology.

/api/v1/prometheus/controller-targets advertises the Controller --metrics-url (KPL_CONTROLLER_METRICS_URL) and returns [] when it is unset. It is a public, read-only endpoint; the URL must be HTTP(S), end in /metrics, and have no credentials, query, fragment, or loopback/unspecified address. The Swarm helper supplies the control node address and KPL_HTTP_PORT automatically.

POST /api/v1/agents/refresh requires a logged-in session and X-KPL-Request: dashboard. It polls registered Agents, including offline records, for current Agent/Peer status. Add ?discover=true to also discover running Swarm Agent tasks and recover missing registrations. The Controller resolves its configured --agent-discovery-dns / KPL_AGENT_DISCOVERY_DNS name and checks each task IP on port 8090. The Swarm stack supplies tasks.<stack>_agent automatically. Discovery uses no Controller Docker socket, task restart, or Peer cleanup.

At most four checks run concurrently within a 15-second overall deadline; DNS has a three-second sublimit, and discovery checks at most 256 unique task addresses. Known addresses are still checked after a DNS failure. Unknown or replacement Agent instances receive an authenticated POST /api/v1/registration/refresh, then must register back with this Controller using their own configured Controller URL and credentials. Reading an unknown Agent's status alone does not register it. Existing identity/generation checks and saved capacity/disabled settings still apply. Already registered instances are checked without restarting history replay.

The response is 200 with {agents, requested, refreshed, failures}. Discovery additionally includes discovery: {enabled, addresses, added, error?}: addresses is the number of unique task addresses found before the limit, and added counts successfully refreshed IDs absent at request start. Successful Agents and failed known IDs are counted once even if multiple addresses are checked; unresolved task addresses are individual failures. Each failure contains {id, name, error}; an unidentified address uses an empty id and its URL as name. DNS/configuration errors appear in discovery.error without discarding successful updates. Unreachable Agent records remain, and normal heartbeat expiry determines offline state. Concurrent refresh/discovery returns 409; an invalid discover value returns 400. Tasks that cannot start or expose their control API need Swarm/Docker startup troubleshooting. Upgrade both Controller and Agents and redeploy the stack configuration to enable registration recovery.

PUT /api/v1/agents/{agentID}/capacity requires a logged-in session and X-KPL-Request: dashboard. The required capacity field accepts a positive integer or null to remove the override; unknown fields, fractions and nonpositive values return 400. It returns the updated Agent with 200, 404 for an unknown ID, or 409 for an Agent that does not report support. Offline registered Agents can be configured. A storage failure returns an error without changing the live setting.

Agent records include defaultCapacity (startup CLI value), optional capacityOverride (saved exception), and capacityPending (awaiting application). capacity is the Controller's currently usable admission limit. Registration and heartbeat responses carry X-KPL-Agent-Capacity and X-KPL-Agent-Capacity-Revision; heartbeat responses remain 204. The Agent applies these values under its admission lock and echoes capacityRevision with its actual capacity in subsequent reports. Only a current acknowledgment permits an increase, so delayed reports cannot undo a reduction. Overrides persist in agent-capacities.json in the Controller data directory; corrupt settings stop Controller startup instead of silently reverting limits.

An Agent returns 429 for a create-node request rejected by its local capacity limit, before creating a Peer. The Controller releases that reservation and waits/retries placement within the existing operation context. Other creation failures retain their usual handling.

PUT /api/v1/agents/{agentID}/enabled sets participation with {"enabled": true} or {"enabled": false} and returns the updated Agent (200). Missing/null/non-boolean or unknown fields return 400; unknown Agents return 404. Login and X-KPL-Request: dashboard are required. Records expose disabled: true when excluded; absent/false means enabled. This is independent of online/offline connection state and capacity overrides.

Disabled Agents remain registered and reachable for existing Peer operations and cleanup. They are excluded from every new placement mode, hardware totals/gauges, Prometheus Agent target discovery and Dashboard topology. Already dispatched creates and existing Peers may finish; disabling does not stop them or rewrite experiment measurements. Explicit/single-Agent joins targeting a disabled Agent wait for reactivation or their existing cancellation/deadline. Full REST inventories retain its Peers; the Dashboard keeps the Agent settings row but omits its Peers and edges. ID-specific settings persist in agent-availability.json in the Controller data directory, and heartbeats cannot overwrite them. Failed saves leave the previous state intact. An unreadable availability file keeps registered Agents disabled until the file is repaired and the Controller restarted.

The browser starts with GET /api/v1/stream?view=dashboard&topology=0. Enabling Overview opens a new connection with topology=1; disabling it returns to topology=0. The first data snapshot on every connection is an event: snapshot containing a complete baseline for the selected dashboard view, followed by event: snapshot_delta updates at most once per second when data changes. Reconnection always starts with a full baseline; no Last-Event-ID replay is required. Idle streams refresh time-dependent values and send event: heartbeat with data: {} if nothing changed after 15 seconds. Unlike SSE comments, this event is visible to the browser’s liveness check.

topology=0 skips detailed node projection and edge construction and sends nodes: [], edges: []. Both dashboard modes include topology: {included, total, ready, starting, other}, with population defined by the topology lifecycle rules. The counts exclude disabled Agents and stopping/stopped/failed Peers and are unaffected by layer/topic filters. Agents, experiment progress, metrics, and event summaries remain available in both modes. The browser shows Transport links as unavailable while Overview is Off or loading. Omitting topology retains graph-enabled behavior for existing dashboard clients.

Snapshot computation is shared, with at most one producer each for the raw view, graph-enabled dashboard view, and dashboard view without graph data. The latter does not wait for an in-progress graph computation. Canceled readers leave immediately; reconnects join the existing producer for their mode. While computation is pending, the server sends a heartbeat every 15 seconds, including before the first snapshot. A heartbeat during the initial wait resets the 30-second connection watchdog without marking the Dashboard Live or replacing displayed data. Metric reconstruction uses a per-run lock, outside the Controller inventory lock used by Agent heartbeats and placement.

Node metadata in the graph-enabled view omits networkRequested and networkSchedule to avoid retransmitting complete plans; it retains the current network, application status and revision. Full REST snapshots retain both plans.

The graph-enabled dashboard view includes live/starting nodes (excluding stopping, stopped and failed nodes), all experiment progress records, topology, metrics, and the latest 40 event summaries. Event summaries contain run/node IDs, type, remote peer, timestamp, latency and the scalar display fields latencyAvailable, direction, controlType, controlEntries, messageIdCount, peerExchangeCount. Raw message-ID/cohort arrays and per-event bandwidth samples are excluded. Full trace records remain available in the ordinary snapshot API and saved experiment exports.

A delta omits unchanged sections. For agents, nodes, and experiments, it contains an object with optional upsert (complete changed/new records keyed by id), remove (IDs), and order (complete ID order when changed). Apply removals, then upserts, then ordering. edges, events, metrics, and topology replace their whole section when present, including empty arrays/null; generatedAt advances with an update. Each connection compares against its own last delivered view, so skipped intermediate updates do not lose changes.

The original GET /api/v1/stream without view retains its full snapshot format, including original recent trace details, for existing clients. Unknown views or nonempty topology values other than 0/1 return 400; a nonempty topology selector also requires view=dashboard. All modes require the same session authentication and stop on logout, session expiry or Controller shutdown. Responses disable caching and request that reverse proxies avoid buffering (X-Accel-Buffering: no).

The Dashboard closes SSE while the document is hidden or the page is leaving, including mobile background tabs. Returning to the page establishes one connection and receives a fresh baseline. The stream remains open during ordinary visible use; its cumulative byte count is not a static page download size.

The browser allows 30 seconds for the initial snapshot and 45 seconds between snapshots, deltas or heartbeats. A timeout closes the stalled connection and enters the same recovery path as an SSE error. The session check is aborted after 8 seconds; a failed/timed-out check still permits retry, while 401 opens the Login required page. Repeated failures back off from approximately 2 seconds to at most 30 seconds with jitter, resetting when data arrives. A browser online event cancels stale connection/probe state and reconnects immediately if visible. Hiding or leaving the page cancels both the stream and any pending session check. On the server, the 10-second write deadline applies only during each write/flush and is cleared afterward, so HTTP/2 idle streams survive until the next heartbeat.

Agent registration, heartbeat, status and Dashboard records optionally carry resources: sampledAt, raw cpuCores, host logical CPU count cpuCapacityCores, memoryUsageBytes, memoryWorkingSetBytes, containers, measuredContainers, complete, and optional error. The scope is the Agent container plus its owned running Peers. Partial samples retain the sums of measured containers with complete: false; zero measured containers has no usable totals. The Controller normalizes sample age and excludes offline or older-than-30-second samples from live sums. Host CPU percent is cpuCores / cpuCapacityCores × 100; unknown capacity yields no CPU percentage but does not hide memory.

New Agent samples also include optional resources.agent and resources.peers objects. Each has cpuCores, memoryUsageBytes, memoryWorkingSetBytes, containers, measuredContainers, and complete, sharing the parent timestamp and CPU capacity. The existing parent values remain Agent + Peers totals. Components have independent availability: a failed Agent sample does not hide measured Peers. An empty, successfully discovered Peer inventory has containers: 0, measuredContainers: 0, complete: true, and zero usage; missing or failed measurements are not zero.

GET /api/v1/agents/resources returns {generatedAt, agents, totals, totalsByScope, services}. Totals retain raw cpuCores and memory fields and include cpuPercent, cpuCapacityCores, cpuMeasuredAgents, measuredAgents, totalAgents, partialAgents, containers, and measuredContainers. cpuPercent weights fresh Agents by host CPU capacity, excluding unknown denominators from both numerator and denominator. Raw core and memory sums can cover more Agents; check the counts. ?format=csv exports per-Agent complete/partial/unavailable status, cpu_percent_host, cpu_capacity_cores, both memory fields and container counts, plus a TOTAL row. Missing values are empty, not zeros. totals remains the combined aggregate; totalsByScope contains agent_and_peers, agent, and peers, each with its own coverage and capacity-weighted CPU denominator. Current CSV adds component rows when reported, and separate TOTAL rows for scopes with valid measurements.

GET /api/v1/agents/resources/history?range=1h retrieves raw Prometheus samples as {from, to, scope, series}; the top-level scope is all, and each series identifies agentId, scope (agent_and_peers, agent, or peers), metric, unit, samples: [{at, value}], mean, min, and max. Add format=csv&kind=samples or format=csv&kind=summary for downloads. CPU is cpu_percent with unit percent_host (entire host = 100%); memory uses bytes and timestamps use UTC. Container coverage is included as containers_expected and containers_measured with unit containers. Combined CPU reads kpl_agent_cpu_usage_percent; component CPU reads kpl_agent_component_cpu_usage_percent. Neither uses legacy raw-core gauges, and each history begins when its corresponding gauge is first collected. The range is 1m–24h. Partial values contribute to statistics without estimating missing usage. Gaps and non-finite samples are excluded, exact duplicates are deduplicated, and conflicting values at one Agent/scope/metric/timestamp fail explicitly. Averages use available scrape samples.

GET /api/v1/agents/resources/measurements returns {generatedAt, maxDurationSeconds, measurements} with newest-first records {id, startedAt, endedAt?, endReason?}. POST to the same path starts a shared interval (201) or returns the existing active interval (200); it does not restart the collector. POST /api/v1/agents/resources/measurements/{id}/stop fixes its end time (200). Repeating a stop keeps the original end time, and stopping an older ID cannot stop the current interval. Mutation endpoints use the same browser-session and CSRF checks as other Dashboard actions. The index is saved atomically in local Controller storage and restored after restart; unreadable or unwritable metadata returns 503 without silently discarding state.

History and interval exports include both existing combined gauges and new kpl_agent_component_* gauges from the same bounded query. CSV preserves its existing column order, appends entity identity fields, and identifies Agent rows using scope; statistics and duplicate checks are independent for each scope. Component series start when upgraded Agents report them; combined-only history is retained without estimating a split.

DELETE /api/v1/agents/resources/measurements/{id} removes one completed interval and returns the updated measurement list (200). Active intervals return 409; unknown/deleted IDs return 404. It uses session/CSRF protection and atomic local persistence; failed writes return 503 without deleting the live record. The corresponding saved-interval export is no longer available. An export that already captured the interval can finish. Shared Prometheus samples and previously downloaded CSVs remain intact.

GET /api/v1/agents/resources/measurements/{id}/export returns the same history JSON as above for the saved interval. Add format=csv&kind=samples or format=csv&kind=summary for downloads. The selection is (startedAt, endedAt] by scrape timestamp, preserving fractional-second boundaries; repeated downloads use the same start and end. Active intervals return 409 until stopped, unknown IDs and empty CSV intervals return 404. Short intervals are supported but may have no samples. Intervals end automatically after 24 hours (endReason: "time_limit"; manual stops use "stopped"). The latest 100 interval records are retained; sample retention remains governed by Prometheus. Exports share the same query limits as recent history, and a failed export leaves the interval available for retry.

All resource endpoints require login. History uses the fixed resource selector at the configured Controller Prometheus URL; callers cannot supply a backend URL or arbitrary PromQL. Exports are bounded to two concurrent upstream queries, 15 seconds, 16 MiB and 250,000 samples. Invalid parameters return 400, no samples for CSV return 404, unavailable/oversized/incomplete Prometheus responses return 502, and a full export queue returns 503. Current CSV needs no Prometheus connection. Resource history is separate from run ZIPs; see KPL resource monitoring.

GET /api/v1/services/resources requires Dashboard login and returns {generatedAt, controllerUptimeSeconds, services}. Service entries contain service, scope, nodeId, nodeName, reportedAt (Controller receipt time), state, optional resources, and optional error. States are running, not_running, unknown, stale, and awaiting; resource sample validity is separate from container running state. Resource fields use the same timestamp/CPU-capacity/memory/coverage structure as combined Agent resources, without Agent/Peer components. /api/v1/agents/resources also includes these entries under services; its Agent totals remain unchanged.

POST /api/v1/services/resources/report accepts internal Bearer authentication only. The body is {nodeId, nodeName, reportedAt, services: [{service, state, resources?}], error?}; report states are running, not_running, or unknown. It is limited to 128 KiB, 64 unique services per node and 512 containers per service. IDs are bounded simple identifiers. Sample ages are normalized to the Controller clock; out-of-order reports are ignored. A cache retains at most 512 nodes, expires hosts after ten minutes without a report, and marks reports stale after 30 seconds. Successful reports return 204, invalid reports 400, missing/invalid authentication 401, and a full node cache 503. Browser sessions cannot submit telemetry.

Resource history and saved-interval exports query the additional kpl_service_* gauges in the same bounded Prometheus request. Each series now includes entityType (agent or service); service series additionally identify service and nodeId, leave agentId empty and use scope=controller|third_party|monitor. running and resource_report_fresh have unit boolean; CPU, memory and coverage retain the existing names/units. Series statistics and duplicate detection distinguish service and host, not only scope. Current/history/summary CSVs append entity_type,service,node_id,node_name after the previous columns; historical node_name is empty. Existing combined/component Agent series remain available. See Controller and supporting services.

POST /api/v1/scenarios/validate accepts a raw YAML body (Content-Type: application/yaml), limited to 1 MiB. A valid scenario returns 200 with {valid: true, name, phases}. Empty input, YAML syntax errors, unknown fields, invalid settings, and multiple YAML documents return 400 with {error: "…"}; parser diagnostics retain line numbers when available. Bodies over the limit return 413. This endpoint uses the same parser as save/run, creates no records or jobs, and reuses the login session. Validation checks configuration, not Agent capacity or runtime connectivity.

All request and response bodies use JSON. List responses omit YAML so opening a large library remains inexpensive; fetch an individual record before editing it.

Reads, validation, and mutations require a login session. Mutations also require X-KPL-Request: dashboard, which the UI supplies automatically. Invalid input, including decoded YAML over 1 MiB, returns 400. The JSON request envelope allows 6 * 1 MiB + 64 KiB to accommodate escaped characters; exceeding that envelope returns 413. Unknown JSON fields and trailing JSON values are rejected. A missing or invalid ID returns 404; storage errors return 500. IDs contain 32 lowercase hexadecimal characters.

Replace control-node:8080 with the Controller address printed by access. First follow API authentication to create KPL_COOKIE_JAR.

curl --fail -b "${KPL_COOKIE_JAR:?Log in first}" http://control-node:8080/api/v1/scenarios

curl --fail -X POST http://control-node:8080/api/v1/scenarios \
  -H 'Content-Type: application/json' \
  -b "${KPL_COOKIE_JAR:?Log in first}" -H 'X-KPL-Request: dashboard' \
  --data-binary @- <<'JSON'
{"name":"Smoke baseline","yaml":"version: 2\nname: smoke-baseline\nphases:\n  - action: stop-all\n"}
JSON

Scenarios and saved results share one set of organizational groups. Each item belongs to at most one group; items without a membership are Ungrouped. These groups organize the library independently of experiment repetition batches, batch notes, scenario peer groups and analysis scopes.

Every successful endpoint returns the complete snapshot {revision, groups, memberships}. groups is an array of {id, name} records; memberships maps item keys to group IDs. Group IDs contain 32 lowercase hexadecimal characters. The initial empty snapshot is {"revision":"0","groups":[],"memberships":{}}. Treat later revisions as opaque strings and send the latest revision with every mutation.

Item keys are scenario:<id> for a saved scenario, batch:<batchId> for an entire repetition batch, and run:<id> for a result without a batch. A batch uses one membership for all its runs, including later appended or retried runs. To remove memberships, move the keys with groupId: "". Moving items and deleting a group apply as a single metadata update. Deleting a group preserves all scenario YAML, runs, batch identities, notes, archives and analyses; its members become Ungrouped.

All endpoints require a login session; mutations also require X-KPL-Request: dashboard. Responses use Cache-Control: no-store. A stale revision or a duplicate name returns 409; reload the snapshot, review the change and explicitly retry. Unknown groups or unavailable items being assigned to a group return 404. Invalid fields, malformed keys, invalid names and group/membership limit violations return 400; unknown JSON fields and trailing values are rejected. Requests over 2 MiB return 413. Storage failures return 500. A failed validation does not partially move items; an uncertain network response should be resolved by reloading before retrying.

Names are trimmed, contain 1–128 valid UTF-8 characters, exclude control characters and must be unique under Unicode case-insensitive comparison. The library supports up to 1,000 groups and 20,000 memberships, with 1–2,000 keys per move request. Metadata persists across Controller restarts in local <data-dir>/library-groups.json, limited to 4 MiB. Group changes do not modify experiment inputs or analysis hashes. An unreadable or invalid metadata file returns an error rather than an empty replacement. See Scenario Library and Results for the UI workflow.

New submissions, retries, continuations and appended sequences share a Controller-wide FIFO queue. 202 confirms admission, not necessarily execution: the first run remains state: "queued" with a zero startedAt until its turn. Multiple submissions may wait. Live experiment objects expose optional queuePosition, the number of earlier submissions (not iterations); zero is omitted. All repetitions of a submission share that position. The position is not saved in result manifests. Failure or cancellation ends only that submission's remaining repetitions; the next submission starts once previous Peer cleanup is confirmed. A cleanup failure holds the next submission with a diagnostic phaseName and automatic retry; stopping that waiter does not release the cleanup barrier. Retained single-run Peers are cleaned up before another submission starts. Queue waiting does not consume phase or schedule time. Restarted Controllers expose reserved unfinished results as interrupted for manual recovery. See experiment queues.

POST /api/v1/experiments returns 202 with the first run's experiment object. Use Content-Type: application/json for {scenario, repetitions, scenarioId?}; scenario is a YAML string, omitted repetitions defaults to 1, and optional scenarioId identifies a saved scenario for group inheritance. Other content types are treated as raw YAML. Raw YAML and the decoded JSON scenario string are limited to 1 MiB. The JSON request envelope allows 6 * 1 MiB + 64 KiB for escaping. An oversized request envelope returns 413; a decoded YAML string over its limit or invalid scenario/repetition returns 400.

scenarioId is the saved scenario's ID of 32 lowercase hexadecimal characters. The Controller executes the supplied scenario text, so it may contain unsaved editor changes. On a new submission it reads the current membership of scenario:<scenarioId> and copies it to batch:<newBatchId>, including when repetitions is 1. A scenario without a group, an omitted ID, or raw YAML creates Ungrouped results. Later moves or deletion of the source scenario do not alter the copied membership. Retry, resume and append retain the existing result membership. Invalid IDs return 400; a saved scenario missing at new admission returns 404. Saved-scenario read errors and group lookup or persistence failures return 500; membership-limit failures return 400. Initial group membership is saved before execution is accepted, so these errors reject admission before the experiment starts.

Idempotency-Key is optional for compatibility, but clients should supply a unique key (up to 128 bytes) and reuse it for the same scenario text, repetition count and scenarioId after an uncertain response. The UI does this automatically using session storage. A duplicate returns the original accepted run without restarting it, including after Controller restart or deletion of its source scenario. It does not reapply group inheritance or overwrite a later manual result move. A key reused for different input, including a different scenarioId, returns 409; if its original result was deleted, 410 prevents recreation. Omitting the key creates a new batch on every successful submission. Back up run-submissions/.

Every accepted initial/retry/resume/append sequence has an executionId. Stop with POST /api/v1/experiments/{id}/stop and header X-KPL-Execution: <executionId> from the current run object; the usual login and X-KPL-Request: dashboard rules also apply. Missing/mismatched identity or a historical member returns 409. A repeated accepted stop for the same execution is harmless. SSE exposes stopRequested so a refreshed UI shows ongoing cancellation.

Example JSON request body for three iterations:

{"scenario":"version: 1\nname: repeat-example\nphases:\n  - action: wait\n    duration: 1s\n","repetitions":3}

Repeated submissions reserve a separate run ID and result record for every iteration, sharing batchId, iteration, and repetitions. Runs execute sequentially; a failed or canceled iteration cancels the queued remainder. Stopping a member of the current execution while its batch is active cancels that sequence. For repetitions > 1, every iteration, including the last, fences and removes its Peers before finalization. A naturally successful single run can retain Peers still running when its scenario ends.

POST /api/v1/result-batches/{batchId}/retry manually restarts saved unfinished iterations in iteration order. For example, if run 4 of 10 failed, completed runs 1–3 stay unchanged and runs 4–10 restart from their first phase. Later completed iterations are also preserved. Failed, interrupted (including never-started records after a restart), and canceled records are eligible. Each new attempt keeps its scenario and seed but receives a fresh ID; previousRunIds in experiment/result metadata lists its earlier attempts. Those logs remain downloadable under Previous attempts; only the latest saved attempt of each iteration contributes to batch progress and the mean. The original /resume endpoint still skips attempted runs and only requeues the never-started remainder.

result-groups/{batchId}/batch.json commits all admission modes, with rollback images for reused continuation IDs. A pre-commit crash rolls back partial reservations; committed unfinished work becomes interrupted. superseded keeps previous attempts out of current progress even if the newest result is deleted. Damaged group records block the affected group and log a recovery error without stopping unrelated Controller startup.

Both continuation modes run on the Controller independently of the submitting HTTP request. They fence and remove prior Peers and refresh Agent state before executing a new iteration. If cleanup fails or the operator stops during cleanup, queued iterations become canceled with the reason in their experiment metadata, and no new iteration starts. Resolve the Agent/cleanup error and retry. Active execution, finalization, downloads and active run/batch analyses block admission; completed run records are never rewritten.

Run/result fields cleanupState (pending, running, complete, failed, retained) and dataState (pending, collecting, complete, incomplete, unverified) are separate from execution state. cleanupError and integrityError preserve problems. A modern Agent advertises runDrain; runs using older Agents are unverified. Detailed results also record agents and controllerVersion; the dashboard SSE omits these environment arrays to keep traffic bounded. run-ownership/ records participants before create RPCs and permits cleanup before Agent re-registration.

Live experiment objects in GET /api/v1/experiments, dashboard snapshots/SSE, and the initial submission response may include timing: estimatedFinishAt (UTC), non-negative remainingSeconds, basis (scenario or observed-runs), observedRuns, and overdue. Active repeated runs also include batchEstimatedFinishAt, batchRemainingSeconds, and batchOverdue (optional zero/false fields). Queued finish times include earlier runs in the same batch. Completed/failed/canceled runs and estimates with insufficient timing information omit timing; it is not persisted in result manifests. See estimated finish times for calculation and uncertainty.

The stop endpoint returns 202 with JSON {"runId":"run-id","status":"stopping"} once cancellation is requested, before cleanup completes. The Dashboard keeps the run or batch controls disabled as Stopping… until a snapshot confirms that no member is running or queued; acceptance does not re-enable the buttons. Observe /api/v1/experiments or the snapshot until the final state is recorded. A run with no remaining cancellation handle returns 404. See Dashboard SSE for compact and legacy stream contracts. /api/v1/events contains at most the 300 most recent events across live Controller state.

From the repository root, replace the host with the Controller address from access and first follow authentication to create KPL_COOKIE_JAR:

curl -X POST http://control-node:8080/api/v1/experiments \
  -H 'Content-Type: application/yaml' \
  -b "${KPL_COOKIE_JAR:?Log in first}" -H 'X-KPL-Request: dashboard' \
  --data-binary @examples/smoke.yaml

The scenario library endpoints store reusable editor inputs independently of experiment results. The list omits YAML, while individual GET, POST, and PUT responses include it. See the scenario library guide for the UI workflow and storage location, and scenario library API for payloads and validation limits.

POST /api/v1/result-batches/{batchId}/append requires the login session, X-KPL-Request: dashboard, and JSON {"additionalRuns":10,"expectedRepetitions":10}. Both fields are positive integers; the resulting total may not exceed 100. A successful response is 202 with the first queued run (iteration: 11, repetitions: 20 in this example). The original batch ID and completed results are retained. Admission uses the Controller lifetime, so closing the request/browser does not cancel execution.

expectedRepetitions must match the complete current group under the admission lock. Concurrent/stale duplicates return 409 instead of appending again. Missing/unfinished/inconsistent iterations, different saved scenarios, active batches, downloads, or analysis also return 409. Invalid bodies/counts return 400, unknown groups 404, insufficient local space 507, and storage errors 500. After an uncertain response, refresh the group before retrying. A committed extension changes the current total and invalidates the old batch membership signature; subsequent batch analysis captures old and new completed runs together.

The local group record is the atomic commit boundary. Pending reservations are hidden and rolled back on recovery; committed queued files remain available for the existing retry workflow. Lists and fresh analysis project the current total without rewriting historical source metadata. Backups must include result-groups/<batchId>/batch.json alongside both result stores.

The results API keeps newest-started-first ordering, with batch/iteration/ID tie breaks for equal start times such as queued runs.

Raw events and observations are recorded locally under <data-dir>/current-run/<run-id>/. The archive worker moves verified log segments and large per-run artifacts to <data-dir>/runs/<run-id>/, retaining a small local metadata index. The default local data directory is data; Swarm mounts it at /var/lib/kpl/data. Only runs/ may be placed on the NAS. See local recording and archive storage for copying, late logs, recovery, and backup requirements.

Use Download results in the Dashboard to export a run as ZIP. Saved results also lists files retained from previous Controller sessions; Refresh reloads that list. Running experiments offer Download snapshot, which contains the records saved when the download starts. These exports include the full saved event log, independently of the 300-event recent buffer. See result downloads for archive contents and collection limits.

GET /api/v1/results exposes saved records. The list field sourceBytes sums the logical source sizes across captured local files and indexed archive objects: scenario.yaml, experiment.json, events.jsonl, observations.jsonl, and optional note.json. It uses local file metadata and the retained archive index without contacting the NAS or reading log contents, reconstructing metrics, or compressing data, so large results remain quick to size. Generated analysis caches/images, the export-time metrics.json and export.json, and filesystem allocation overhead are excluded. Unreadable experiment metadata can still have a source size. If the source files cannot be statted safely, the field is omitted and the UI shows Source · —.

DELETE /api/v1/results/{id} returns 204 after local deletion and tombstone persistence (NAS file removal is asynchronous), 404 if the result is absent, and 409 while the run/batch is active or an actual GET download holds it. List reads do not cause a download conflict. A short pin while opening remote source descriptors can return 409; a HEAD calculation releases that pin after capturing descriptors.

DELETE /api/v1/result-batches/{batchId} uses the login session and exact batch ID. It checks every member before deleting any files, returns 409 if a run is active/finalizing or a ZIP download holds a member, and removes all saved member runs (including failed/canceled runs), their individual analyses, and the separate batch mean. Success returns 200 with {"deletedIds":["run-id", "..."]}; an absent group returns 404. An orphan batch mean can also be removed. Storage failures return 500 and may leave a partially deleted group; refresh the list and resolve the storage error before retrying. The browser uses a 30-second request timeout; the Controller may finish after that timeout.

The dashboard makes no automatic size HEAD requests and runs no ZIP-size expiry timers. An explicitly requested HEAD /api/v1/experiments/RUN_ID/download still measures the exact ZIP Content-Length with a streaming byte counter; individual GET downloads use that measurement too. Bulk downloads stream without a size measurement. Archives are not retained wholesale in memory or on disk. Concurrent measurements for one run are shared, with at most two different runs measured at once. For API compatibility, the list can still return an already prepared, valid downloadBytes value, but the UI does not use it.

When no publication is pending, an unchanged file/state version keeps the measured byte count while each HEAD or download records a fresh, full-precision export time; the stored export.json entry uses a fixed-width timestamp so its encoded length stays constant. Stable list rows omit downloadSizeMaxAgeMs, and stable HEAD responses omit X-KPL-Result-Size-Max-Age-Ms. A result with pending publications briefly reuses both its byte count and export boundary. Its list row includes downloadSizeMaxAgeMs, and HEAD includes X-KPL-Result-Size-Max-Age-Ms; each value is a positive number of milliseconds calculated from the server's remaining cache lifetime after subtracting a one-second response safety grace. This relative lifetime avoids depending on synchronized client and server clocks. The first list that observes a prepared value extends the short lifetime once, then reports the max age from that extended boundary so an immediate GET still matches the displayed size; repeated list refreshes do not keep sliding the deadline. If the server has no lifetime left beyond the safety grace, the list omits both downloadBytes and downloadSizeMaxAgeMs, while HEAD measures a new boundary before responding.

The first HEAD after Controller startup or a source/state change must read the captured files and rebuild metrics; pending results repeat that work after their short cache expires. HTTP request cancellation is checked while waiting for a measurement slot, reading source logs, and aggregating the reconstructed metrics; canceled requests stop without producing a server-error response. Large saved event logs can therefore take longer to measure. This work occurs only for explicitly requested HEADs and actual downloads; displaying source sizes does not require it.

Reconstructing ZIP metrics uses memory proportional to distinct event IDs and message/receiver pairs; the raw file copy itself is streamed.

POST /api/v1/downloads accepts {"kind":"scenarios","ids":["scenario-id"]} or {"kind":"results","ids":["run-id"]}. Result IDs must identify individual Runs, including batch members; the dashboard expands selected batches before submission. The request requires the normal login session and X-KPL-Request: dashboard. Select 1–2,000 IDs per request, with a 512 KiB JSON body limit; duplicate IDs are removed. Validation reads local metadata without opening archived source logs.

Success returns 201 with {"url":"/api/v1/downloads/<ticket>","count":1,"expiresAt":"<UTC timestamp>"}. An invalid selection returns 400, missing items 404, unreadable metadata 500, and a full ticket store 429. Tickets expire after 10 minutes, are held only in memory, and are invalidated by Controller restart. At most 16 pending tickets are retained.

An authenticated GET to the returned URL consumes the ticket and streams scenarios-<UTC timestamp>.zip or results-<UTC timestamp>.zip as an attachment. No complete ZIP is retained in browser JavaScript or on the server, and no preliminary ZIP-size pass is performed. Up to two bulk transfers may run at once; additional GETs return 429 with Retry-After: 5 without consuming the ticket. An expired, consumed or unknown ticket returns 410. Download URLs do not replace authentication. HEAD is not supported.

Scenario entries contain unchanged saved YAML. Result entries contain individual Run ZIPs, each captured when processing reaches that Run. Download tickets do not reserve every selected source: deletion or storage failure before or during processing may fail the transfer. Before ZIP output starts, missing items return 404 and read failures 500; afterward the server aborts the stream and leaves no complete outer ZIP. Cancellation releases the current Run's download lease and stream slots. Scenario downloads and result downloads describe filenames and contents.

GET /api/v1/result-storage requires login and returns localDirectory, archiveDirectory, availableBytes, minFreeBytes, checking, checkStartedAt, lastCheckedAt, phase, lastProgressAt, stalled, resultsRevision, and optional error. Space is measured on the local Controller data filesystem; archive health is the last background-worker observation, so the request never waits for a NAS probe. phase distinguishes preparing, discovering, copying, verifying, publishing, deleting, paused (while experiments run), and idle. The server sets stalled only while an archive I/O phase has reported no progress for more than 15 seconds; total cycle duration, local preparation, and deliberate pauses do not trigger it. Copy and verification update progress for each completed chunk, and resuming after a pause resets the timer. The UI uses this server decision, so browser clock differences cannot create a false warning. This is a lack-of-progress indicator, not a NAS benchmark or an automatic cancellation. The dashboard checks this small endpoint every 15 seconds while visible, independently of SSE. resultsRevision changes when storage updates become visible and across Controller restarts; the UI compares this opaque string to refresh Saved results only when needed.

Saved-result rows include storage: {state, error?, updatedAt} with local, pending, archiving or archived. These fields do not change experiment state or batch-mean membership. sourceBytes includes archived log segments and local tails once each. Archived source downloads and analyses require the archive to respond; list/note operations use the local index. Results stored before this feature become visible as the startup/periodic background index discovers them.

A fresh POST /api/v1/experiments returns 507 when local free space is below the configured threshold. Repeated and resumed batches can remain queued with phaseName: "Waiting for local result storage" before the next run; cancellation remains available. The standard CLI default is 1 GiB via --run-min-free-bytes / KPL_RUN_MIN_FREE_BYTES; zero disables free-space checks. Active runs check space and recording errors about once per second; failure stops the run with incomplete data and cancels its queued remainder. The threshold does not reserve enough space for an arbitrarily large active run.

GET /api/v1/results/{id}/note returns {runId, text, revision, updatedAt?} with Cache-Control: no-store. A run with no note returns empty text, revision "0", and no timestamp. Both reading and writing require a login session. PUT also requires X-KPL-Request: dashboard and JSON {text, revision} using the revision returned by the last read. On success it returns 200 with the saved record and a new opaque revision when the text changes. updatedAt is a UTC RFC 3339 timestamp. A stale revision returns 409; fetch again and review the current text before resubmitting. Notes are shared, not private to a session.

Text is limited to 16,384 decoded UTF-8 bytes. Whitespace-only text clears the note; otherwise whitespace and line breaks are preserved. Missing/null fields, unknown fields, trailing JSON and oversized decoded text return 400. The JSON envelope permits 6 * 16,384 + 4,096 bytes to accommodate escaped text and metadata; larger requests return 413. Invalid or absent run IDs, including deleted results, return 404. Storage errors return 500 without silently replacing an unreadable note. An unchanged save keeps the revision and timestamp.

The Controller atomically replaces local current-run/<run-id>/note.json under the same persistence lock as run deletion. The archive worker mirrors it to runs/<run-id>/note.json outside that lock. A cleared note retains its revision to prevent stale edits from restoring removed text. Existing results need no migration, and no experiment metadata or analysis cache is rewritten. Notes can be edited during runs or downloads; an already captured ZIP retains its captured note version. GET /api/v1/results includes optional note: {preview, updatedAt} for nonempty notes, with whitespace collapsed and the preview limited to 160 Unicode code points plus an ellipsis. Fetch the note endpoint for the full text. Notes are absent from dashboard SSE.

GET / PUT /api/v1/result-batches/{batchId}/note use the same limits, revisions, status codes and session/CSRF rules, returning {batchId, text, revision, updatedAt?}. The batch must have saved members or an existing group metadata directory. Group notes are local result-groups/<batchId>/note.json records: retries, member deletion and analysis refreshes do not overwrite them. Full group deletion removes the note after all member deletions succeed; an orphan note can also be removed through the group deletion API. Stale editors cannot recreate a deleted group.

The results list includes groupNote: {preview, updatedAt} on at most one member per group to avoid repeating the same preview for every run; clients group by batchId before displaying it. Full group notes load only on request and are absent from SSE and individual run ZIPs. Back up result-groups/ with local Controller data; group notes are not mirrored into NAS run archives.

Use POST /api/v1/analysis-jobs/{id} to analyze one saved run, then poll GET on that path. Both POST and GET/HEAD require a login session. An admitted queued/running job returns 202; reuse of a current completed artifact returns 200. At most 32 queued/running jobs are admitted across the Controller (503 when full), and the shared analysis slot permits one run analysis at a time. Closing a client does not cancel admitted work.

Within a run, graph metrics for independent observation snapshots are calculated in parallel. The worker count is min(4, max(1, GOMAXPROCS - 1)), leaving one Go execution lane available on multi-CPU Controllers. Identical graphs share in-flight and cached calculations (up to 256 retained entries). Results keep their original observation order, and phase: graph metrics completed/total counts completed observations. An error or cancellation stops the remaining workers before the analysis finishes; metric definitions and cached analysis versions are unchanged. Individual, batch and synchronous analyses all use this calculation path.

The status state is idle, queued, running, completed, failed, canceled, or interrupted. idle means the saved run has no requested analysis. Failed/canceled/interrupted work can be resubmitted. Persisted unfinished work recovered after restart becomes interrupted; orderly cancellation can already have saved canceled. There is no separate job-cancel endpoint. Deleting an eligible saved run cancels its job and removes its artifacts.

During phase: saving, an individual or batch job remains running; status reads, experiment event ingestion and stop requests can continue while the result files are saved. Before publishing completion, the server rechecks that the attempt is current and uncanceled and, for an individual analysis, that its source has not been deleted.

The server also marks a completed artifact stale: true when its computation version differs from the current analyzer, even if the sources are unchanged. This applies to individual and batch status and Saved results listings. Clients use that marker to request an update without maintaining their own analyzer-version threshold. The saved artifact remains downloadable until replacement.

Requests reuse an existing queued/running job even with ?refresh=1. For completed work, sourceRevision is only a cheap change hint; list/status requests do not read NAS log bodies. Controller-managed archival preserves that revision for unchanged source content, including while sealing local logs; see source changes. Changed revisions leave individual GET status completed with stale: true. A normal POST verifies sourceHash (a SHA-256 fingerprint of sampled logical source bytes) in the bounded worker. Equal hashes and a matching analysisVersion restore the original completed job ID, artifact and snapshot timestamps, update its revision and expose reused: true. Consequently, polling a verification job can finish with the original artifact’s ID. Different hashes, outdated computation versions or legacy artifacts without hashes require computation. Images performs these checks automatically; GET alone does not start work. Reanalyze sends ?refresh=1 and bypasses completed-artifact hash reuse. A fresh analysis cannot restore absent historical evidence.

The fingerprint format is sampled-sha256-v1:<hex>. For each logical source (scenario.yaml, experiment.json, events.jsonl, observations.jsonl), it includes the name, total byte length, sample offsets and sampled bytes. Files up to 64 KiB are read in full; larger files use 128 windows of 512 bytes, including both ends and pseudorandom interior positions. Positions are reproducible from the algorithm version, name and length, so independent snapshots and different archive segment layouts compare the same bytes. The worker reads at most 64 KiB per source through random access; verification byte progress counts those reads. Equal fingerprints can miss same-size edits outside the samples. They are an approximate cache check, not full-file integrity verification. Analysis itself still reads all required inputs, and ?refresh=1 forces that calculation. Existing full-hash artifacts remain cached while their source revision is unchanged; when that revision changes, they need one calculation to establish a sampled baseline. Archive-copy integrity and security/build hashes continue to use full contents.

The result dialog displays a saved analysis while source hashes are compared or a replacement is computed, for both individual runs and batch means. It keeps the saved snapshot timestamp and marks verification progress separately. Equal hashes keep the displayed charts and filters; a completed replacement updates them. Verification errors leave the saved view available with an error message. Manual reanalysis also keeps the current view while waiting.

Pending or failed replacement jobs expose savedAnalysisId when an earlier artifact is still available. Add &saved=1 and that explicit jobId to the existing /result (or individual /summary) endpoint to read it. Ordinary requests still reject unfinished attempts with 409. The saved reference survives Controller restart and artifact publication failures; it is cleared only when the completed replacement metadata is published; it is not a permanent archive of analysis versions. A mismatched/obsolete ID returns 409. Client downloads remain pinned to the analysis currently displayed.

After completion, GET or HEAD /api/v1/analysis-jobs/{id}/result?jobId={jobId} serves the full JSON; /summary?jobId={jobId} serves the comparison artifact. Supplying jobId guards against downloading a different attempt; omission selects the current completed attempt. An unfinished/mismatched attempt returns 409, a missing run 404, and malformed/unreadable stored data normally 422. The compact artifact sets observations, scoreTimeline, timeline, bandwidthTimeline and research.messages to empty arrays while preserving messageCount, metrics, aggregates, distributions and fits. It cannot supply per-message paths or original timelines. Both responses carry analysisId, analysisVersion, and the source asOf boundary.

scoreTimeline preserves detailed score components independently of general observation downsampling. Its group/point fields, bounds and missing-data rules are defined in score component timelines. It is retained in full individual and batch artifacts, and empty in the individual comparison summary.

research.receiverGroups preserves the receiving peer group (group; empty means unknown/conflicting metadata), eligible population, message count, summary statistics, compact overview, propagation/duplicate/hop/eager/lazy curves, and primary latencyCDF/latencyHistogram. Latency distributions retain the main metric definition; other group curves retain the research definition. The same group data survives batch compaction. New trace events carry optional group; legacy analysis recovers identities from all saved graph snapshots before display downsampling. See receiver-group semantics.

GET /api/v1/experiments/{id}/analysis remains a synchronous compatibility route with a two-minute request timeout. It computes a response without creating a persisted background job. Use the job API for long analysis and later downloads. The metric guide owns research definitions; monitoring owns saved files, and visualization describes the browser workflow.

POST /api/v1/batch-analysis-jobs/{batchId} analyzes completed runs from the same batch, including appended runs. All batch members must finish execution/cleanup and at least two must be completed. Active batches return 409; insufficient completed runs or inconsistent metadata return 422; an unknown batch returns 404. Authentication, duplicate admission, retry and restart behavior follow individual analysis. Individual and batch jobs share the 32-job admission bound; up to 100 repetitions are processed sequentially within one batch job.

Status identifies the batch with batchId (runId is empty) and adds selected runIds, completedRuns, totalRuns, expectedRuns, and a membership signature. Source verification runs first, reading one captured run at a time under the shared worker limit; byte fields describe the current phase/run. Calculation progress weights each run equally. snapshotAt is the most recently analyzed run boundary; runs[].asOf records individual boundaries. Membership, state or source-revision changes expose the previous completed cache as idle with stale: true. A normal POST compares a batch sourceHash covering completed-run sample fingerprints, member identities/outcomes and historical attempt accounting. Equal hashes reuse the original batch artifact and ID; changed inputs recompute. Excluded event logs and notes are not metric inputs. Local source revisions include file identity/change time where available, so preserved-size/mtime rewrites still request verification; archived revisions use manifest metadata without reading NAS bodies. NAS objects are immutable and must be changed through the storage workflow. ?refresh=1 forces fresh computation. If batch membership or source revisions change during verification, calculation or saving, the job fails; retry to capture the current sources.

The membership signature uses source revisions only for current completed contributors. Excluded and superseded attempts still contribute their identities and outcomes; changes to their unused logs do not queue source verification for the batch mean.

reliability in the batch artifact provides attempts, failed, canceled, interrupted, retries, failedAttemptRate, unknownAttempts, unverifiedRuns, incompleteRuns, environments, environmentChanged and warnings. Counts include recorded started attempts and retained deleted-attempt outcomes, not never-started queued records. Failure rate is failed / attempts; the metric mean remains over current completed runs. Each environment has a fingerprint, run IDs, Agent snapshots and Controller build identity. Host load and unrecorded network differences are outside this fingerprint.

The artifact has version: 1, analysisVersion: 8, aggregation: "equal-run-mean-v1", batch/attempt identities, expectedRuns, missingRuns, excluded, runs, and summary. Summary entries under metrics.*, research.*, bandwidth.sentBytes and bandwidth.receivedBytes contain average, deviation (between-run sample SD), median, and count (valid runs). An undefined mean or single-run SD is null. Compact runs preserve graph metrics, observations, bandwidth, distributions, and message time series/receiver curves/origin counts in research.overview; raw message/node paths are omitted. See batch means for chart alignment and distribution semantics.

Files are persisted as <data-dir>/batch-analyses/{batchId}/job.json and result.json. Closing a view or restarting the server does not remove a completed result. Download with an explicit jobId selects that completed snapshot, returning 409 if replaced by another attempt. Deleting an individual source does not remove the existing batch snapshot files. Reanalysis replaces the current files with the changed selection; download an older result before refreshing if it must be retained.

The supplied Swarm stack enables this path through Controller --network-manager (KPL_NETWORK_MANAGER_URL) and Agent --auto-reset-network --control-network NAME. Custom deployments that omit these options retain the previous network lifecycle. The stack places control services on a stable overlay and runs network-manager with the control manager's Docker socket and a persistent recovery volume. Its standalone gateway has no Docker socket or host mounts. See deployment and upgrade requirements.

Before each run, the Controller sends POST /api/v1/network/fence to every registered Agent, including disabled Agents. The body is {runId, epoch, requestedAt}. Each Agent closes new admission only after existing processes have confirmed removal, then checks Docker for remaining managed Peer containers. A refused or unreachable Agent blocks network preparation. The Controller then sends POST /api/v1/network/prepare to the manager with the same identity and an agents map of Agent IDs to control URLs. The manager serializes preparation, rejects older requests, requires Ready Swarm nodes, validates network ownership/default IPv4 settings and service references, and preserves the subnet and gateway while creating a new network ID. Network deletion is never forced. Its local recovery files retain the allocation if deletion succeeds but creation or acknowledgment fails.

After a successful Docker removal response, the manager waits until the exact Peer network name is absent from the combined network list before creating its replacement. Removal visibility, retries of the specific network with name ... already exists rejection, and visibility of a successfully created network share a one-minute limit within the existing manager budget. Polls are cancellable; an inspection failure is never treated as confirmed absence. A different ID appearing during removal blocks preparation and is not deleted.

A failed create response is reconciled through inspection. The manager may continue only when a new network has the current epoch, expected ownership and allocation, no endpoints, and no service references. Otherwise it retries only the explicit name-conflict rejection after checking state again; arbitrary failures do not trigger another create. The original network ID is never accepted as the replacement. Missing visibility, persistent conflicts, cancellation or unverifiable state prevent activation. These waits do not advance the experiment or scenario schedule clock.

The response is {runId, epoch, networkId, networkName, gatewayUrl, peerGatewayUrl}. The Controller sends it to each Agent's POST /api/v1/network/activate; only after all acknowledge does it set the run's startedAt and begin phases. Experiment.peerNetworkId records the new identity. Each CreateNodeRequest.peerNetworkId must match the Agent's current run/epoch admission, and Docker creation uses the immutable ID. A worker can materialize its overlay on its first container attachment; the Agent checks the actual network ID after startup. A stale create cannot use a replacement merely because it has the same name. Preparation failure starts no phases and follows normal failed-run cleanup/queue handling.

These preparation endpoints require internal Bearer authentication. Each endpoint accepts one JSON value and rejects unknown fields; Agent bodies are limited to 64 KiB and manager bodies to 1 MiB. Agents return 204 on confirmation, 400 for malformed input and 409 when preparation, cleanup or activation cannot be confirmed. The manager returns the JSON acknowledgment on 200, malformed input on 400, and rejected/failed preparation on 409; authentication failures return 401. The preparation budget is four minutes, including manager work bounded to three minutes and Agent calls bounded to 45 seconds. The gateway forwards authenticated HTTP control/telemetry and the public Controller clock-health request between overlays; Peer P2P traffic stays direct. Gateway resources belong to control-plane monitoring. Gateway replacement is deferred until the next confirmed cleanup barrier. The path does not stop services, restart Docker or change data volumes. Production connectivity and repeated large-run behavior require deployment validation.

These REST/JSON endpoints serve component communication. Registration, heartbeats, and forwarded events are sent to the Controller; lifecycle commands are sent to an Agent.

CreateNodeRequest.experimentStartedAt is an RFC3339 timestamp copied from the current execution's startedAt and persisted in the Peer process configuration. It is required when config.network.schedule has nonempty changes with reference: experiment-start; omission is rejected before creating the Peer. The Controller fills it automatically. It is an internal request field, not a scenario YAML field. Static network settings and peer-relative schedules do not require it.

Agents advertising createReplay: true support recovery of interrupted POST /api/v1/nodes responses. The Controller includes CreateNodeRequest.agentStartedAt, copied from that Agent's startedAt; a different instance returns 409 before admission. For the same node ID and identical decoded request, concurrent requests wait for the original admission and later requests return the existing Peer status, including terminal status from local history. Different request contents or an unsuccessful original admission return 409. The Agent retains a SHA-256 request fingerprint and admission outcome in its existing process/history records (metadata.createRequestHash and metadata.createAdmission); it does not keep a separate growing request cache. These fields are Agent-owned. Replay never restarts a Peer or resets its lifetime. Requests omitting agentStartedAt keep the legacy duplicate-ID rejection.

For capable Agents, EOF, truncated responses, connection reset/refusal, broken pipe and request timeouts permit at most three attempts on the same Agent URL/instance, with 250 ms and 500 ms waits. Each attempt retains the normal 10-second HTTP limit and the scenario context can shorten all work. The effective profile, seed, lifetime and node ID remain fixed through recovery; the experiment clock continues normally. HTTP refusals and invalid acknowledgments are not transport retries. Older Agents receive no new field and no automatic creation replay. If a dispatched request remains uncertain, its reservation stays occupied until the Agent reports the Peer or acknowledges the matching run-generation fence and completed cleanup. The Controller does not re-place that uncertain request on another Agent or treat an empty inventory as proof that a delayed creation cannot arrive.

For scheduled Peers, node metadata.networkRequested retains the requested network configuration, and metadata.networkSchedule contains the Agent-resolved plan, including concrete seeded delays. These metadata values are JSON strings. metadata.network contains the current concrete configuration; treat it as provisional while networkPending is "true". After a successful Peer application, status reports set networkPending: "false", networkRevision (a decimal string; 0 for the initial state, then the one-based change index), and networkAppliedAt (RFC3339). A late join or delayed timer may skip revisions. The Agent rejects regressions to older revisions, and heartbeats propagate the applied state to the Controller. These fields are reported state, not an API for changing a running Peer's configuration.

Periodic Agent heartbeats use partial: true and split Peer records into JSON bodies of at most 10 MiB. If one record alone exceeds the limit, the Agent reports an error naming that node and leaves it unacknowledged. Live Peers, unresolved container removals, and terminal Peers awaiting Controller acknowledgement remain in memory. Confirmed removals carry metadata.cleanupComplete="true"; this includes failed Peer processes whose containers were successfully removed. Their final state/error remains intact while old topology and score data are released.

After acknowledgement, a background worker retains the 64 most recent terminal records and commits older compact records in bounded transactions to one local bbolt file, <agent-data-dir>/peer-history/nodes, before removing them from memory. Periodic heartbeats omit acknowledged terminal records. GET /api/v1/status and GET /api/v1/nodes return the current inventory plus retained recent/pending records, not every historical Peer. A full status still reconciles missing active nodes; it does not rewrite a confirmed cleaned failure as a success. Historical records remain in Controller results and local Agent history.

Re-registration replays retained records and starts a bounded disk-history replay after each live heartbeat: at most 128 records and normally 1 MiB per page (one larger record can use the existing heartbeat limit). Failed pages are retried without rebuilding the full in-memory process map. Temporary transport failures do not trigger a new full replay. Replay resumes by a copied database key; no read transaction survives a network send. Retired node IDs remain unusable through indexed disk lookup, including after Agent restart. Legacy JSON histories are automatically converted before startup reconciliation, and their verified backup files are cleaned incrementally. Converted histories require a compatible Agent version; see history migration. Run-generation fences beyond a 128-entry memory cache persist under peer-history/fences/; an older stop cannot lower a stored fence. Stop-all also fences runs already retired from memory.

New admission returns 503 with Retry-After: 1 when 4,096 completed records await acknowledgement/local retirement, or 4,096 termination markers await event-queue admission. Existing Peers and their pending evidence remain intact; failed Docker cleanup continues to consume capacity. History storage I/O runs outside the Agent state lock, and a fence/history revision check prevents retirement or stop requests racing with disk lookup from allowing stale admission.

Telemetry requests are limited to 5000 events and a 10 MiB JSON body. Peer and Agent senders split batches by their encoded byte size, including escaping and the envelope; retries retain source order and event identities. A successful prefix is removed before sending the remaining events. Both receivers validate every event before admission: a single event that cannot fit in a 10 MiB encoded batch returns 413, with none of that request admitted. The Agent checks after normalizing its identity. This also prevents direct Controller submissions from creating oversized event lines that archive analysis cannot read.

The Controller admits at most two telemetry requests through decoding and durable commit. Up to 64 additional requests wait without reading their bodies for at most one second; cancellation releases a waiting request. Saturation returns 503 with Retry-After: 1 and no acknowledgment, so the Agent keeps and retries the batch. A request already committing may finish after its client times out; event identities still prevent retry double-counting. These request limits bound concurrent decoded bodies, not total Go heap usage.

Agents persist accepted telemetry batches in <agent-data-dir>/telemetry-spool/ before acknowledgment. A durable prefix cursor permits replay after restart; Controller event batches are synchronized to local disk before acknowledgment. Agent replacement must retain its data volume. Delivery is at least once, so event identities remain unchanged; this cannot recover events never accepted from a Peer or a lost volume.

Spool recovery retains a compact offset/run index and reads a forwarding window of at most 5,000 events / 1 MiB of serialized events (one individually valid large event may exceed the window). Telemetry decoding/admission allows two concurrent requests, with at most 64 additional undecoded waiters for up to one second. Busy decoders and batches that would exceed the 50,000-event / 256 MiB pending backlog return 503 with Retry-After; already-accepted disk/in-flight events are preserved. A full backlog also pauses new Peer admission. Terminal evidence from already-running Peers may temporarily exceed the cap, and legacy larger backlogs drain without being discarded. Run drain includes matching events still on disk.

For POST /api/v1/nodes, a pre-admission refusal carries X-KPL-Admission-Retry: not-created and Retry-After. The Controller releases the reservation, temporarily excludes that Agent, and retries within the scenario context. Unpinned joins can use another eligible Agent; pinned/single-agent joins retain their target. The two legacy telemetry/terminal-evidence backlog errors with Retry-After are also recognized during rolling upgrades. Other 503s/timeouts are not automatically re-placed.

New spool writes coalesce into 4 MiB-target .jsonl segments and append checksummed ACK journal records; existing .json arrays remain recoverable. The format-v2.json guard blocks older Agents from ignoring pending new-format events. A drained normal shutdown removes the guard and idle segment.

During orderly Agent shutdown or DELETE /api/v1/nodes, telemetry drain waits for Controller acknowledgement of both queued events and any batch already in flight, including final Peer events. A failed batch retains its order for retry; drain reports failure if acknowledgement cannot complete within its time limit.

JSON bodies must contain one value. Trailing whitespace is allowed within the body limit; a second JSON value, trailing garbage, or a body over the decoder limit returns 400. Generic Controller/Agent JSON handlers use a 10 MiB limit; Peer /publish uses 1 MiB and /mesh-freeze uses 4 KiB. Scenario request envelopes have the separate limits described above. A locally generated Peer event that cannot be JSON-encoded or fit in a batch is logged and counted in telemetry_drop, preserving its source sequence gap while later events continue. Network failures retain the pending batch for retry.

Peer /publish has a 10-second publication timeout, including waiting behind another local publication. Request cancellation or deadline expiry releases queued requests without waiting for the current publisher and returns 504 while waiting for the gate/readiness. Queued canceled requests do not publish or emit a successful publish event; publication timestamps still start after the gate and readiness wait.

The Agent additionally exposes this Controller-driven cleanup endpoint:

Internal endpoints may change independently of the operator API. Request and snapshot field definitions are in internal/model/model.go, with bandwidth types in internal/model/bandwidth.go; handlers are in internal/controller/api.go and internal/agent/api.go.

With scoring and inspection enabled, Peer status includes scoreSample: {observedAt, sequence, intervalSeconds, components}. Every component has {count, mean, min, max} over that observer's valid scored-peer snapshots. Keys are p1, p2, p3, p3b, p4, p5, p6, p7, topicCap, and total. Agent reports clone these snapshots and reject older component observations. Full node/network/snapshot REST responses retain them; Dashboard SSE omits component summaries. peer_score events carry the same scoreSample in events.jsonl, including source identity and event sequence, but do not fill the recent network-event feed. Existing per-target peerScores totals remain available. These are summaries, not a saved matrix of individual scored-peer/topic pairs.

The optional sample sequence is a positive inspection counter within one Peer process, independent of the enclosing event's sequence. Agent uses it to accept newer observations, including empty snapshots, after a backwards Controller clock-offset correction. observedAt retains the corrected snapshot time. Legacy samples with absent/zero sequence are ordered by observedAt until a sequenced sample arrives; they cannot replace a sequenced sample. A new Peer process starts a new Agent-managed status lifetime.

The scenario v3 mesh-freeze action sends POST /api/v1/nodes/{nodeId}/mesh-freeze to each target's Agent using internal Bearer authentication. The Agent forwards the same JSON to the Peer's POST /mesh-freeze with its internal token and X-KPL-Node-ID: {nodeId}. Request and successful 200 response:

{"runId":"run-id","generation":1}

{"nodeId":"node-id","peerId":"actual-libp2p-peer-id","frozen":true}

The Agent verifies readiness, enabled GossipSub, the meshFreeze capability, exact run/generation, and run fences before forwarding. It validates the returned node/Peer identity and rechecks the same process and scope before accepting the acknowledgment. The Peer also requires matching node identity, run, and generation. These checks prevent a delayed request or reused container address from freezing a different Peer. Success means the local PubSub event loop applied the one-way, idempotent freeze; it does not establish simultaneous application across Peers. Protocol behavior defines which topology and message operations continue.

The scenario timeout covers dispatch and acknowledgments for the whole selection. Each Agent-to-Peer request and Peer handler retains a ten-second limit; increasing the scenario timeout does not extend those individual limits. Cancellation propagates through the control requests and releases event-loop waits.

The Agent returns 400 for malformed input or a missing runId, and 502 when target validation, forwarding, or acknowledgment validation fails. The Peer rejects malformed/unknown JSON fields with 400, identity or configuration mismatches with 409, shutdown or cancellation with 503, and its ten-second application deadline with 504. Both endpoints return 401 for missing/invalid internal authentication, require POST, and return 405 for other methods. A failed request can leave a Peer frozen if application preceded a lost response; retries are idempotent, with no rollback or unfreeze.

Node metadata exposes string values meshFreezeEnabled for the configured capability and meshFrozen for confirmed application. Once true, later status reports cannot reset meshFrozen. On its first confirmed freeze the Peer emits a mesh_freeze event with fields.frozen: true, scope: "peer", and membership: "pinned"; repeated successful commands do not duplicate that event. This follows normal telemetry delivery and loss boundaries. Payload types are in internal/model/mesh_freeze.go.

Scenario v3 topology sends POST /api/v1/nodes/{nodeId}/topology to the Agent, which forwards to Peer POST /topology. Internal Bearer authentication, X-KPL-Node-ID, exact run/generation, readiness and run fences follow the mesh-freeze checks. Both endpoints require one JSON value, reject unknown fields, and limit requests and acknowledgments to 1 MiB.

{
  "runId": "run-id",
  "generation": 0,
  "topologyId": "controller-generated-sha256",
  "stage": "prepare",
  "topic": "kpl/graph",
  "neighbors": [{"peerId": "actual-libp2p-peer-id", "addresses": ["/ip4/10.0.0.2/tcp/20000"]}]
}

prepare validates all input, establishes required connections and waits for matching PubSub subscriptions. It stores one prepared plan without installing its mesh. After all targets prepare, apply with the same plan ID, topic and neighbor set installs the exact topic mesh and freezes all local topic meshes. The topologyId identifies the complete graph and Peer incarnation; it is shared across the domain. Self/duplicate neighbors and invalid or mismatched /p2p address suffixes are rejected. A Peer permits at most 9,999 neighbors, each with 1–32 addresses of at most 2,048 bytes.

A successful 200 response has {nodeId, peerId, topologyId, stage, topic, frozen, neighbors}; neighbors is the exact sorted Peer ID list. An apply acknowledgment requires frozen: true. Both Agent and Controller validate identity and the neighbor set, then recheck the selected incarnation. The Peer operation has a 30-second limit; Agent→Peer requests allow 35 seconds and Controller→Agent requests allow 40 seconds so downstream timeout responses can arrive. The overall scenario timeout can shorten these limits. An already applied identical plan remains valid if explicitly submitted again after a lost response, including when a physical link has since disappeared. Different applied plans conflict. A successful prepare reserves no remote state, so apply rechecks current eligibility.

The Controller permits at most three attempts per Peer preparation command (the first request plus two retries), and only for transient HTTP 503/504, network failures or an individual request timeout. It waits 250 ms and then 500 ms before retrying; waiting stops on cancellation. Each attempt revalidates the selected Peer incarnation and sends the same plan. All attempts share the overall phase deadline. Validation errors, permanent conflicts such as Peer 409, and invalid acknowledgments are not retried. The Controller starts apply only after every exact-plan prepare acknowledgment, and never automatically retries apply.

During preparation, each Peer connects to at most four neighbors concurrently. Transient dial failures retry within the existing command budget, with waits increasing from 250 ms to at most 2 seconds. If the first attempt encounters libp2p dial backoff, preparation clears that selected neighbor's backoff once before retrying; backoff from later attempts remains in effect. Other peers and global dial settings are unchanged. Retry waits stop on cancellation, and configured transport dial timeouts still apply. A later backoff does not replace an earlier transport failure in the Peer timeout diagnostic. Retries cannot make an unreachable peer or a connection/resource limit disappear.

Agent returns 400 for malformed input and 502 for failed target validation, permanent Peer errors or invalid acknowledgments. It preserves transient Peer 503/504 responses, reports transient forwarding failures as 503, and reports its request timeout as 504. Peer returns 400 for malformed fields, 409 for identity, capability or plan conflicts, 503 for cancellation and 504 for timeout. Both require internal authentication (401) and POST (405 otherwise). Controller errors from the prepare stage state that apply was not started. A canceled or failed apply may already have taken effect; no distributed rollback is provided.

Controller topology_generated records model, parameters, effective seed, generator version and graph size. topology_assignment records intended neighbors and optional coordinates per node with evidence: "planned". topology_stage records prepare/apply acknowledgment counts plus the fields below. Peer topology_applied records the locally confirmed plan ID, topic and neighbors once, subject to normal telemetry delivery/drop reporting. Full adjacency stays in saved logs and is omitted from the recent-event feed. Active edges continue to come from runtime mesh snapshots. Types are in internal/model/topology_command.go.

Scenario v3 profile schedules send POST /api/v1/nodes/{nodeId}/profile to the Agent, which forwards to Peer POST /profile. Both require internal authentication, POST and one JSON value, reject unknown fields, and bound requests/acknowledgments to 32 KiB. The Agent includes X-KPL-Node-ID; both hops check the exact run, generation, readiness and Peer incarnation, and the Agent checks run fences before and after forwarding.

The request is {runId, generation, revision, stage, set}. revision is a positive integer up to 2147483647, increasing within a run generation; stage is prepare or apply. The typed set fields are defined by the scenario reference. A 200 response is {nodeId, peerId, revision, stage, effective}. effective contains complete concrete values for the requested sections; both Agent and Controller check the acknowledged requested values and identities. Prepare validates and retains one bounded pending update without applying it. Apply requires the same prepared revision and patch. A successful identical request may be acknowledged again; stale revisions and conflicting same-revision values fail. New Peers inherit the run's latest profile revision and matching patches, so older delayed commands cannot revert their startup settings.

The Controller prepares the full cohort before any apply and does not retry apply automatically. Local network changes and PubSub updates are separate operations; a late failure can leave part of a Peer update applied. The Peer reports this failure and rejects subsequent profile commands. Network operations have a 5-second sub-limit; the Peer request allows 8 seconds, Agent forwarding 10 seconds and Controller forwarding 12 seconds, all shortened by cancellation/the per-change timeout. Peer errors use 400 for malformed input, 409 for conflicts or application errors, 503 for cancellation and 504 for deadlines. Agent validation/forwarding/acknowledgment errors use 502; malformed input uses 400. Authentication and method failures use 401/405.

Confirmed Peer status exposes profileRevision, runtimeProfile and profileAppliedAt; Agent acknowledgments merge changed sections and a newer Peer status can recover a lost acknowledgment. An uncertain apply sets Agent profilePending: true; uncertain network application also sets networkPending: true until confirmed. Prior confirmed configuration remains historical evidence, not proof of the failed command's outcome. Future-shaping admissions carry internal networkMutable: true to grant container NET_ADMIN; initial inherited revisions use profileRevision. The configuration events retain requested versus confirmed settings and partial failures.

Updated Peers add metadata actually exposed by upstream libp2p to events.jsonl. Existing publish/deliver/duplicate events and main Metrics aggregation remain in use. messageId is the envelope application ID or the raw pubsub-<hex native ID>; fields.pubsubMessageId and detailed ID lists encode pubsub wire IDs as hex.

Detailed IDs are limited to 8,192 entries and 512KiB of hex per category; subscription lists are capped at 8,192. RPC/entry/ID totals retain their full counts even when detailed lists are truncated. Existing telemetry_drop and source sequences expose event loss. Payload bodies, IWANT sender-queue causes and nonexistent global RPC IDs are not collected. Eager/Lazy follows the metadata estimation rules in experiment metrics.

The hopwave* fields are unsigned wire reports. Their interpretation and compatibility requirements follow HopWave; existing analysis estimates are unchanged. Logs without wire metadata acquire no HopWave fields.

Set the same KPL_USER and KPL_PASSWORD on the Controller and every Agent. Both are required at startup. The dashboard first opens Log in; a successful login creates a random, server-side session identified by an HttpOnly, SameSite=Strict cookie. The cookie is Secure over HTTPS (including a trusted reverse proxy setting X-Forwarded-Proto: https). Sessions expire after 12 hours, are revoked by Log out, and do not survive a Controller restart. Logging out or expiry closes that session's SSE streams without stopping experiments or background analysis. There is one configured account with full dashboard permissions.

All dashboard/API reads, writes, SSE, scenario validation, analysis and downloads require a session. Unauthenticated API requests return JSON 401; browser GET/HEAD navigation that requests HTML receives a styled Login required page with the same 401 status. The page is also available at /login-required and links to the login form and dashboard. Dashboard API/SSE session expiry opens this page once and stops reconnecting; explicit logout returns directly to /login. Opening the dashboard without a session still redirects to /login. Session-authenticated mutations also require X-KPL-Request: dashboard and reject cross-origin requests. The UI adds this header automatically; it never asks for another token and stores neither the password nor the session cookie in localStorage. Login accepts only JSON {user, password}; wrong credentials return 401, and ten failed attempts from one connection IP within five minutes cause 429. Sessions and failure records are bounded to 1,024 entries each. Login response writes do not hold the session lock, so a stalled login client does not block other session checks or logout.

Only the login/error-page assets, health/clock endpoint, /metrics, and the two Prometheus target-discovery endpoints remain public. Internal Agent registration/heartbeats/events and Peer bootstrap/discovery accept an automatically derived, domain-separated service key. It is generated from the account configuration and passed to Peers without exposing the login password to them. This key cannot access dashboard result APIs or start experiments. KPL_API_TOKEN and the --token CLI flag are retired; direct service clients must be updated together. Changing credentials requires redeploying Controller and Agents, with no active experiments or retained Peers using the old key. Grafana keeps its separate login.

For REST clients, replace the address below with access output and log in once. The cookie jar is private and can be reused by the examples above:

KPL_URL=http://control-node:8080
KPL_COOKIE_JAR=$(mktemp)
read -r -p 'Username: ' KPL_USER
read -r -s -p 'Password: ' KPL_PASSWORD; printf '\n'
export KPL_USER KPL_PASSWORD
python3 -c 'import json,os; print(json.dumps({"user":os.environ["KPL_USER"],"password":os.environ["KPL_PASSWORD"]}))' |
  curl --fail-with-body -c "$KPL_COOKIE_JAR" \
    -H 'Content-Type: application/json' --data-binary @- "$KPL_URL/api/v1/auth/login"
unset KPL_PASSWORD
curl --fail-with-body -b "$KPL_COOKIE_JAR" "$KPL_URL/api/v1/snapshot"

After completing your requests, log out and remove the cookie jar:

curl --fail-with-body -b "$KPL_COOKIE_JAR" -H 'X-KPL-Request: dashboard' \
  -X POST "$KPL_URL/api/v1/auth/logout"
rm -f "$KPL_COOKIE_JAR"

HTTP does not encrypt credentials or cookies. Use HTTPS or a trusted management network; an HTTPS proxy must overwrite the forwarded protocol header.

The same four scenario job counters are present in /api/v1/snapshot and SSE snapshots. The dashboard displays them on each run, so active, successful, failed, and canceled background work is visible without inspecting Controller logs.

See visualization for analysis responses, charts and export formats.

Bandwidth measurement · v2 analysis coverage audit

Korean translation: 한국어.