Wiki home

Documentation index · Source repository

This guide owns saved source files, downloads, deletion, and backup boundaries. Use execution and recovery for continuation/retry, visualization for figures and comparisons, and metrics for formulas.

Saved results supports searching by experiment name, batch ID, or run ID, together with In progress, Completed, and Needs attention status filters. A matching run keeps its whole batch visible, including previous attempts, so progress and batch actions remain available. Status filters use current attempts: Completed requires every current run in the batch to be complete; Needs attention includes failed, interrupted, canceled, or unreadable results.

The displayed count is the number of visible saved runs. Filters stay selected while the result list refreshes. Choose a status directly in the selection bar; All removes only the status filter. The search field’s Clear search button removes only the search text. To restore the full list, clear the search, choose All, and set the group filter to All groups. The status choices stay visible, in two rows on narrow screens, and support keyboard arrow keys. On mobile, result rows become cards with labeled times. Each run keeps Images and its download visible; More contains available recovery and deletion actions. Batch headers keep Analyze batch mean visible beside their More menu. The Agent table remains horizontally scrollable.

Within a repetition batch in Saved results, rows are ordered by numeric Run n of M (1, 2, …, 10), including queued runs and retries. The order stays stable when a queued run starts; previous attempts remain in their separate table and entries without a valid iteration appear last. Batches keep their archive position.

Use Groups to manage the same groups available in Saved scenarios, then Select → Move to organize existing results. The group filter is shared between the two views. Follow the shared group workflow for creation, renaming, selection, moving to Ungrouped, deletion and refresh behavior.

New results from a selected saved scenario inherit its group when the submission is accepted, including a single run and unsaved editor changes. New unsaved inputs and ungrouped saved scenarios produce Ungrouped results. Later moves or deletion of the source scenario leave existing results in place. Retry, resume and Add runs retain the result set’s current group. See the scenario group workflow for details.

The group dropdown counts result sets: an entire repetition batch counts as one, and a standalone run counts as one. The result-list summary counts saved runs. A batch's previous attempts, retries and added runs share its membership. Group changes affect organization only; they do not change batch membership, source files, analysis hashes or batch means. Delete group preserves the results under Ungrouped; More → Delete batch deletes the batch's results after confirmation.

In Saved results, choose Select, check runs or batches, and choose Download ZIP. Select visible respects the group, search and status filters. Selecting a batch includes all of its currently listed runs, including failed runs and previous attempts. The selected Run IDs are fixed when Download ZIP is pressed; runs added afterward require another download.

The outer ZIP contains one [experiment name]-<run-id>.zip per Run. Each inner ZIP has the same contents as the individual result export. Download chart PNG/CSV ZIPs from the analysis window, or use the CSV-only selection export below.

Runs are captured and streamed one at a time, so each has its own snapshot boundary; active runs remain partial snapshots. A missing or unreadable item stops the download. If transmission has begun, the browser receives an interrupted transfer rather than a completed archive with missing items. Refresh the list and retry after resolving the item or storage issue. Only the Run currently being read is protected from deletion. See the bulk download API for preparation, expiry and concurrency limits.

Download CSV ZIP in selection mode saves results-csv-<UTC timestamp>.zip. It contains one [experiment name]-<run-id>-csv.zip per selected Run. Each inner ZIP contains only the overview chart CSVs used by Images, across every category and protocol; PNGs, chart-definition JSON and source logs are excluded. Batch selection includes its currently listed individual Runs, including previous attempts, and does not add a batch mean. Use the batch analysis window to export its mean separately.

The CSV export reuses each Run's latest available completed analysis, including while verification or replacement is in progress. It does not force reanalysis; use Reanalyze first if a fresh snapshot is required. If no completed analysis is available, it submits or waits for background analysis. Runs are processed sequentially with progress and Cancel. Cancel stops browser preparation; accepted server analysis continues. Selected IDs are fixed when pressed, selection remains after completion or failure, and an unreadable/replaced analysis prevents a partial archive from being downloaded. The browser creates this ZIP with a 128 MiB combined archive limit; select fewer results if exceeded. This differs from the streamed source ZIP and is not a hard limit on total browser memory. Per-Run snapshots need not share a computation time.

Choose Add note or Edit note beside a run's name in Saved results. A short preview appears in its row; the editor shows the complete plain-text note and its last saved time. Notes are shared by logged-in viewers, work for queued/running/finished results, and support up to 16 KiB of UTF-8 text. Select Save note to persist changes, or leave the text empty and save to clear it. Cancel discards the current draft.

Each run ID has its own note, including previous attempts in a repetition batch. Retrying an experiment creates a new run ID and does not copy or overwrite the old attempt's note. Notes survive Controller restarts and are saved separately in current-run/<run-id>/note.json and mirrored to the archive, so experiment finalization cannot replace them. The result ZIP includes that file once a note has been saved; clearing keeps an empty record to protect against stale edits. Result deletion also deletes its note, and backups must include local pending notes as well as archived copies.

Choose Add batch note or Edit batch note in a batch header to annotate the whole repetition batch, even while it is collapsed. Batch notes use the same editor and conflict protection, but are separate from each run's note. They remain attached to the batch ID across retries and individual-run deletion. Delete batch removes the batch note after all member deletions succeed. Batch notes are stored locally in result-groups/<batch-id>/note.json; include that directory in Controller backups. They remain editable with the NAS offline and are not included in an individual run ZIP.

If another window changes the note, saving reports a conflict and keeps your draft. Choose Review latest note to display the current saved version alongside your draft, then edit and save again if you want to replace it. Storage errors also keep the draft open. Lists carry only a bounded preview; full notes load when the editor opens and are not added to the dashboard SSE stream. See the note API.

Completed batches expose More → Add runs to append more repetitions to the same batch. For example, 10 completed + 10 additional runs becomes a 20-run batch, and Analyze batch mean recomputes statistics over all 20 after completion. See adding runs for eligibility, limits, and recovery.

In the Dashboard, choose Download results on an experiment or in Saved results. For repeated runs, expand the series header to access each run's download, analysis and deletion controls. The saved list reads the Controller's data directory, so results remain accessible after a Controller restart. Use Refresh after restoring files from a backup. Running experiments offer Download snapshot. Sizes immediately show Source, the sum of uncompressed original files at the last list refresh. Running and queued results show Live source; refresh the list to update it. This value can differ from the downloaded ZIP size.

Each ZIP contains:

The export captures local file boundaries and immutable archive references under the Controller's persistence lock and streams the ZIP after releasing that lock. Events appended later are excluded, so a slow download does not hold up telemetry writes. A completed run can still receive delayed telemetry: download again after collection has settled if you need those later records. partial: false indicates a terminal recorded run state, not a guarantee that no telemetry was lost. The archive does not contain message payloads, PCAP files, or the Prometheus/Grafana databases.

Source / Live source excludes analysis caches, generated images, export-time derived files, and filesystem overhead. The Dashboard does not measure ZIP size automatically. Download API details define sourceBytes, explicit HEAD requests, and cache headers.

Saved-result lists and downloads require login. After API login, reuse the cookie jar:

curl --fail -b "${KPL_COOKIE_JAR:?Log in first}" http://control-node:8080/api/v1/results
curl --fail -b "${KPL_COOKIE_JAR:?Log in first}" --output run-results.zip \
  http://control-node:8080/api/v1/experiments/RUN_ID/download

Replace RUN_ID with an ID from the saved list. Use the control node address printed by access.

Exports contain collected telemetry, including subscription-session start/checkpoint/stop evidence and recorded Agent termination confirmations. Peer retries, Agent queue backpressure, and graceful drains reduce loss but have finite bounds; forced termination does not drain the Peer. Source sequence gaps make missing receipts unknown. A download cannot recover missing events or reveal completely invisible sessions. /api/v1/events still returns only the latest 300 events, and the web event view displays the newest 40 from that buffer, while the ZIP reads the full saved log.

Saved results → More → Delete permanently deletes the selected run's scenario, metadata, notes, events, and live metric index after confirmation. Running/queued experiments, members of an active batch, and results being downloaded are protected. Deletion first synchronizes a tombstone and its parent directories to local storage, then removes local records and their in-memory archive bookkeeping. The archive worker removes NAS files afterward, retrying if unavailable. Successful local deletion changes the storage revision so other open dashboards refresh their saved lists even while the NAS is unavailable. The deletion API is DELETE /api/v1/results/{id} with the login session. It does not stop Peers; previously scraped Prometheus/Grafana history remains. Deletion markers prevent late telemetry from recreating a deleted result; preserve them with Controller data in backups and migrations. See the REST API guide for status codes and download headers.

List inspections do not block deletion. An explicit ZIP size request (HEAD) briefly protects an archived run while opening NAS source files; deletion can return 409 during that step. After capture, size calculation does not retain the deletion guard. Actual ZIP downloads (GET) protect their captured result until the request finishes. The delete request has a 30-second browser timeout; on timeout the Controller may still finish, so refresh or retry the same result. A slow follow-up list refresh does not keep the dialog controls disabled.

The series header's More → Delete batch confirms the saved-run count and deletes the entire batch through DELETE /api/v1/result-batches/{batchId}. Unlike individual-source deletion, this also removes the separate batch-analyses/{batchId} mean files and cancels pending analyses. All members are checked for activity/downloads before deletion starts. A storage error can leave partial progress; refresh and retry after resolving the error. Existing Prometheus/Grafana history remains.

A run's raw export and its computed analysis have different boundaries. Download results / Download snapshot captures the source files when exporting. Download analysis JSON uses the boundary recorded by the completed analysis job; logs appended later do not enter that artifact until a fresh analysis runs. Compare export.json.exportedAt with analysis asOf/job snapshotAt when checking totals.

Server analysis files are not included in Download results ZIPs. Use the analysis download endpoints to retain computed JSON, and save chart downloads separately. A comparison ZIP records the generated charts, not the full source archives or imported files needed to rerun the analysis. Keep those sources and the selected Series/Case/parameters with the software revision.

Completed analysis survives Controller restart in the data volume. A new attempt replaces the current cache; download it first if an older analysis boundary must be retained. Accepted server calculations continue after closing the browser; image conversion and comparison calculations run when the view is reopened/recreated. A browser download is not a server-rendered image archive. Failed/interrupted/canceled jobs can be retried, as defined in the background API. Deleting an eligible saved run removes these analysis files with its source records.

Repetition Analyze batch mean jobs are stored separately in <data-dir>/batch-analyses/{batchId}/job.json with result.json and result-next.json as publication slots. Their artifacts contain equal-run means, sample SDs, valid counts and compact inputs for reproducing overview charts. Individual-source deletion leaves an existing batch snapshot on disk; reanalysis after membership changes replaces the current batch files. Download analysis JSON or the mean PNG/CSV ZIP, and include current-run, runs, batch-analyses, and result-groups in server backups. See batch means for eligibility, exclusions and sample-count semantics. Backups must preserve both slots and their job metadata.

Individual and batch analyses write replacements into the inactive slot and synchronize each file and its directory before atomically publishing the completed job metadata. The previous saved-analysis reference remains readable if artifact publication fails, including when the full analysis was written but its compact summary failed. Job metadata selects one matching full/summary pair; interrupted replacement files cannot be served under the previous analysis ID. Legacy metadata selects the original filenames. On recovery, job metadata must contain exactly one JSON document and a valid slot; trailing non-whitespace data or an invalid slot is a storage error and cannot authorize serving a cached artifact.

Analysis reuse compares sampled fingerprints of scenario.yaml, experiment.json, events.jsonl and observations.jsonl. A cheap source revision detects potential changes without reading NAS logs during list/status requests. Images · Check and Batch mean · Check verify changed hints in a bounded background worker. Matching samples and analysis versions reuse the original artifact and snapshot, including after timestamp changes. This can miss same-size edits outside the sampled positions; Reanalyze always reads the inputs and computes again. See the sampling contract for the byte limit, position selection and hash format. Batch fingerprints also account for membership and attempt outcomes; excluded runs’ unused logs and notes do not invalidate their means. Existing full-hash artifacts stay cached while their source revision is unchanged; a later revision change establishes a sampled baseline through one calculation. Artifacts without a hash also need a baseline. Existing completed analyses and downloads remain available while checking or replacing them.

Controller-managed log sealing and Local → Archived moves preserve the source revision when only the file layout changes, so reopening an unchanged individual or batch analysis needs no extra NAS log hash pass. The local archive manifest records the matching layout fingerprint and original revision; it remains valid after restart only while that fingerprint matches. Late telemetry or source replacements invalidate it, including same-size rewrites with restored mtime. An interrupted move or restored legacy archive can require one normal hash check. Saving identical experiment metadata keeps its existing file and does not queue another archive or verification.

Mean metrics and contributing run counts includes started attempts, failures, cancellations, interruptions and retries, as well as warnings about incomplete evidence and changed execution environments. Failure rate is failed attempts divided by all recorded started attempts; never-started queued records are excluded. The success-only mean still uses current completed iterations. Deleting a known attempt retains its outcome in batch metadata; outcomes deleted before this accounting existed are reported as unknown.

Environment groups retain Controller/Agent build identity, actual Peer image identity when available, Agent instances and capacities. Legacy missing records are explicit. These records do not capture all host-load or network changes, so inspect differing environments before combining results.

During an active run, Recording experiment data… and Cleaning up Peers and collecting final logs… use neutral text. Queued runs do not show a redundant cleanup-pending warning. A completed single run that retains Peers shows Peers retained · Recording experiment data…. Cleanup failures, incomplete or unverified data, and interrupted collection still use warnings; fully collected and cleaned runs omit the extra line.

The Controller treats runs/ as potentially slow external storage. New runs record scenario, state, telemetry, observations and notes in <data-dir>/current-run/<run-id>/ on local storage. Only mount runs/ on the NAS; keep the data directory itself and current-run/ local and persistent. Existing NAS results are discovered in the background at startup and approximately once per minute, without copying their large logs back onto local storage. Initial discovery can take time; the storage status line distinguishes discovery from an empty archive.

A single archive worker copies eligible results to runs/, synchronizes and checks their SHA-256 digests, publishes an archive manifest, and then removes copied local logs and large per-run analysis files. Initial archives use a hidden .incoming-<run-id> directory before publication. A small local index remains in current-run/: scenario, experiment metadata, note, analysis job status, and .kpl-archive.json. This intentional metadata retention makes lists, notes, deletion and recovery decisions independent of NAS latency. Run notes are saved locally immediately and mirrored later. Batch notes remain in the separate local result-groups/ directory.

Copying and hashing also require the complete captured file length. An early end of file, including a source truncated after capture, fails the transfer before publication or local cleanup; matching checksums of a shortened prefix do not certify a complete archive.

The worker leaves active/queued runs, runs with known live Peers, and active analysis jobs local. It waits for a five-second quiet period on both open logs before sealing either one, and pauses copying and verification between 256 KiB chunks while an experiment is running. Each background transfer/verification pass is limited to 32 MiB/s while idle; both passes must complete before local bytes are removed, so total archive throughput is lower. Already issued filesystem operations cannot necessarily be canceled, and explicit downloads/analysis can still generate NAS traffic; this does not provide physical network isolation when the NAS and experiment share a NIC.

Logs are sealed into immutable segments before transfer. Late telemetry is always appended locally, even after the earlier log was archived. Result downloads and analysis read the archived segments plus the captured local tail as one logical log, without double counting a segment left behind by an interrupted cleanup. Use the download/analysis API for a complete result: after late telemetry, the NAS events.jsonl alone may be only the first segment. Preserve .kpl-archive.json and all segment/object files when copying archive directories manually.

Local, Archive pending, Archiving and Archived describe storage separately from experimental success/failure. A NAS outage preserves local data and does not stop local recording or metadata operations. Archived source downloads and analyses still need the NAS. A blocked mount can occupy the single archive worker or either of two bounded NAS readers, but no NAS filesystem operation holds the Controller's experiment persistence lock. Unfinished transfers are retried after restart; a restarted experiment itself remains interrupted and requires the existing continuation/retry workflow.

Saved results automatically follows storage changes through the lightweight storage-status poll (normally every 15 seconds while the page is visible), even after all experiments and analysis jobs have finished. A changed storage revision refreshes the local result index; an unchanged revision does not repeatedly reload the full list. Returning to the page checks again. Changes arriving during a list request are fetched afterward, and a failed refresh retries on a later storage poll without clearing the displayed results. Late notes or telemetry can move an archived result back to Archive pending until the new local data is archived. Byte-identical metadata rewrites update the manifest's local file timestamps without re-uploading the source files.

The storage status line distinguishes copying, verification, finalization, archive discovery/deletion and Archiving paused while experiments are running. These are normal activity states. It warns when an archive I/O step has no reported progress for over 15 seconds, rather than when the entire worker cycle takes 15 seconds. Existing transfer limits and experiment pauses still apply.

During a run, local space and recording failures are checked about once per second. Falling below the threshold or a log/metadata write error fails the active run, triggers cleanup and cancels its queued remainder. Incomplete data is explicitly marked; failed metadata writes are retried once storage recovers. This is a guard, not a hard quota: a burst can fill the disk between checks, and filesystem I/O can still stall. The next run waits for enough local space.

The CLI and Swarm deployment default to requiring 1 GiB free locally before starting a run. Set --run-min-free-bytes or KPL_RUN_MIN_FREE_BYTES (for example sh scripts/swarm.sh configure KPL_RUN_MIN_FREE_BYTES=2147483648), then redeploy; 0 disables free-space checks. A new submission below the threshold receives 507. A repetition batch waits before its next run and remains cancelable; archival can free space while it waits. This is an admission threshold, not a reservation or a per-run size limit: provision enough local space for the largest run, analysis output, and any NAS outage backlog.

After the first successful connection, .archive-store-id locally and runs/.kpl-store-id identify the archive destination. If the mount disappears or a different destination appears, copying is deferred instead of filling the directory beneath a missing mount. Ensure the intended NAS is mounted before the first initialization, and preserve both identifiers in backups. Changing archive destinations requires an intentional storage migration, not deleting these markers while the Controller is running.

In Swarm with KPL_STACK_NAME=kpl, the kpl_controller-data volume on the control node is mounted at /var/lib/kpl/data. Live files and the retained metadata index are under current-run/<run-id>; archived objects are under runs/<run-id>. Ordinary service recreation and swarm.sh remove preserve the volume. There is no automatic raw-event retention limit or cross-node replication.

Preserve the entire Controller data directory, including hidden entries:

If group assignment is interrupted, Controller startup checks the local batch commit record. It preserves the current membership of an accepted batch and rolls back the pending assignment of a batch that was not accepted. It then removes library-group-admission.json; later manual moves or group deletion are preserved. An unreadable recovery or batch record stops startup with an error so the Controller does not guess which assignment to keep.

A run ZIP preserves one experiment, not the entire Controller. Back up Prometheus and Grafana volumes separately for their history/settings; see monitoring retention. Restoring run files makes results available; it does not restore sessions, live counters, or execute unfinished runs automatically. Use recovery deliberately. See deployment storage and restart for volume location, write durability, and shutdown conditions.

Korean translation: 한국어.