Wiki home

Documentation index · Source repository

This guide describes the version 3 YAML format, Peer profiles, protocol controls, network conditions, and background jobs. For saving reusable YAML in the Dashboard, see the scenario library.

Scenario format versions are independent of K-P2PLab release versions. Use version: 3 for new scenarios; it adds mesh-freeze, model-based topology formation and their group/ID selectors. The parser still accepts versions 1 and 2 for existing scenarios; omitted or zero version resolves to 1. Existing actions retain their behavior, while the two mesh control actions require version 3. Unknown YAML field names are rejected.

The executable schema and validation are in internal/scenario/scenario.go; scheduling is in internal/controller/runner.go.

role controls Kademlia bootstrap discovery: a boot node is advertised by the Controller as a bootstrap peer, while a worker node is not. Topic transport discovery is separate and considers ready PubSub participants in the same run and exact topic. type controls the node's libp2p, Kademlia, and PubSub behavior. These concepts are independent, so an experiment can use several worker behaviors without abusing the bootstrap role. When both type and profile are omitted, role: boot selects the boot preset and every other role selects full.

Reusable node profiles belong in the top-level profiles map. A join phase resolves its configuration in this order: built-in type, named profile, then the phase's inline node overrides. If the phase omits type, the named profile's type chooses the preset; otherwise the role supplies the default. A profile must name a defined profile or a built-in preset; unknown names fail validation even when type is explicitly set. Pointer-backed boolean and numeric controls preserve explicit false and 0, which allows experiments such as disabling lazy gossip with historyGossip: 0. This does not extend to legacy flat numeric fields, whose zero values mean omitted; use the nested controls for explicit zero overrides.

The built-in boot type sets gossipsub.enabled: false. Set it explicitly to true in a profile or inline node block when bootstrap nodes should also participate in PubSub. Once enabled, boot nodes can use the router-appropriate parameters, scoring, and inspection options described below; examples/mixed-workers.yaml demonstrates this opt-in. PubSub-enabled Peers poll the Controller's same-run, exact-topic registry immediately after startup, then on a three-second loop. Each Peer ranks the full eligible set with rendezvous hashing and fills the deficit between its current topic peers and DHigh, subject to its total connection budget and retry backoff. See bootstrap and topic discovery for timing and fallback behavior. GossipSub still forms the actual GRAFT mesh; DHT bootstrap, transport candidacy, and mesh membership remain distinct.

version: 2
name: mixed-workers
seed: 42
onExit: cancel
jobShutdownTimeout: 3m

profiles:
  tuned-mesh:
    type: full
    libp2p:
      connectionLimit: 128
      connectionManager:
        lowWater: 64
        highWater: 96
        gracePeriod: 30s
    kademlia:
      mode: server
      protocolPrefix: /k-p2p-lab/v3
      bucketSize: 20
      concurrency: 10
    gossipsub:
      router: gossipsub
      topics: [kpl/default]
      floodPublish: false
      peerExchange: true
      params:
        d: 8
        dLow: 6
        dHigh: 12
        dOut: 3
        dLazy: 8
        heartbeatInterval: 500ms

phases:
  - action: join
    group: tuned
    role: worker
    profile: tuned-mesh
    count: 100
    parallel: true
    parallelism: 16

The nested node configuration exposes scalar parameters and named callback/interface policies supported by the pinned libp2p packages. The complete protocol reference lists every mesh, score, PeerGater, filter, validation, discovery, publication, and Kademlia field; see also the protocol example. Go duration strings such as 250ms, 30s, and 5m are used for every duration field.

GossipSub-only controls (params, hopwave, meshFreeze, active score/inspection, floodPublish, peerExchange, directPeers, and peerGater) apply only to the gossipsub router. FloodSub and RandomSub reject active scoring but can retain an inactive tuning block.

The base mesh defaults are D=6, DLow=5, DHigh=12, DScore=4, DOut=2, DLazy=6, history 5/3, gossip factor 0.25, and a 1s heartbeat. When PubSub is enabled on boot, that preset uses DScore=3. full/worker and the role-focused GossipSub worker presets inherit the v2-derived DLow=5, DScore=3, maxIHaveLength=5500, and 1s initial heartbeat. Most worker presets also use the v2 hard connection limit of 55; light uses 32. Kademlia defaults to bucket size 20 and protocol prefix /k-p2p-lab/v3. See internal/model/config.go for resolved defaults and the protocol reference source map for the split typed schema.

libp2p.connectionLimit is only a resource-manager hard cap on total connections; it does not implicitly create a soft connection manager. The soft low-water/high-water trimming behavior is installed only when libp2p.connectionManager is explicitly present, using its lowWater, highWater, and gracePeriod. This lets a profile choose the hard cap, the soft manager, or both independently.

When migrating a v2 configuration, map v2's misleadingly named protocol_id to v3 kademlia.protocolPrefix: v2 used that value as a prefix. In v3, kademlia.protocolId instead overrides the exact Kademlia V1 wire protocol ID; it is mutually exclusive with protocolPrefix and protocolExtension, so do not copy a v2 prefix into it. The default prefix /k-p2p-lab/v3 is an intentional new protocol namespace. Use protocolPrefix: /k-p2p-lab/kad-dht only when an experiment requires the legacy v2 namespace.

For RandomSub, randomDegree sets the minimum connection/degree target through libp2p's process-global RandomSubD. KPL isolates each Peer in its own container, so that global applies to one Peer. randomNetworkSize is the separate estimated network-size argument passed to NewRandomSub.

Scoring is disabled by default. Only gossipsub.score.enabled: true enables it; a score block or inspection interval alone does not. When explicitly enabled, scoreInspectInterval defaults to 1s, while 0s disables inspection. Each inspection updates the node's peerScores map. It is returned by /api/v1/nodes, /api/v1/network, /api/v1/snapshot, and the SSE snapshot stream; selecting a node in the topology displays the observed score count and average in its details.

appSpecificScore supplies P5 through a fixed default value and optional peers map keyed by libp2p Peer ID; explicit peer values, including zero, override the default. A nonzero active appSpecificWeight requires that policy. All values and weighted P5 results must be finite. Active selective validation safely defaults omitted decay groups to 1s/0.01 and an omitted disabled P1 quantum to 1s; explicit zero ticker/divisor durations are rejected. Profile-to-inline score overrides replace the complete score block and preserve explicit zero/false values. See scoring configuration.

Add network to a profile or a join phase's node block. The default scope: p2p shapes outgoing P2P TCP on port 20000 inside each Peer container. Select scope: all to reproduce v2's shaping of all egress, including control HTTP and telemetry. A configured delay is a one-way egress delay, not a round-trip latency target. rateMbps and tbf.rateMbps configure egress limits; bandwidth measurement separately records actual libp2p stream usage.

node:
  network:
    delay: 100ms
    jitter: 10ms
    lossPercent: 1
    duplicatePercent: 0.1
    corruptPercent: 0.1
    reorderPercent: 1
    rateMbps: 10
    queueLimit: 1000

See the v2 reproduction audit and mapping and churn example. They cover placement (balanced, per-node random, batch single-agent, or explicit agentId), onError: continue for churn, payloadEncoding: raw for exact PubSub data length, and topic: '*' for all topics. Network scope and payload encoding retain their existing defaults. Worker bootstrap now uses seeded first-success selection, and the transport stack is explicitly TCP/Noise/Yamux.

Peers with network conditions require Linux NET_ADMIN and host-kernel sch_netem support. The Docker runtime also grants NET_ADMIN to joins that can receive a version 3 schedule network change, even before the first impairment. Missing kernel support or a failed tc command fails node startup explicitly.

examples/network-conditions.yaml creates two bootstrap nodes and four constrained workers, waits for initialization, publishes sample messages, and stops all nodes. Run it from the dashboard or submit it with:

Replace control-node:8080 with the Controller address printed by access. First follow API authentication to create KPL_COOKIE_JAR.

curl -X POST http://control-node:8080/api/v1/experiments \
  -H 'Content-Type: application/yaml' \
  -b "${KPL_COOKIE_JAR:?Log in first}" -H 'X-KPL-Request: dashboard' \
  --data-binary @examples/network-conditions.yaml

Both version 2 and version 3 accept network.schedule in profiles and inline node settings. Existing static settings remain the initial conditions. To change the delay from 10 ms to 100 ms ten minutes after the current experiment starts, and restore it at fifteen minutes:

network:
  delay: 10ms
  schedule:
    reference: experiment-start
    changes:
      - after: 10m
        set:
          delay: 100ms
      - after: 15m
        set:
          delay: 10ms

To give every Peer its own countdown, change reference to peer-join (the default when omitted). For example, start at 10 ms and change to 100 ms three minutes after each Peer begins initialization:

network:
  delay: 10ms
  schedule:
    reference: peer-join
    changes:
      - after: 3m
        set:
          delay: 100ms

Explicit values such as delay: 0s, lossPercent: 0, and rateMbps: 0 disable the corresponding inherited condition. If a positive delay had jitter or reordering, clear those dependent settings in the same patch before removing the delay. Setting a new delay replaces an inherited delayDistribution, and vice versa. Distributions are resolved by the Agent using the per-Peer seed; a new explicit distribution is sampled once for that change, while an inherited sampled delay is retained.

A profile override replaces the whole schedule, rather than appending changes. Use schedule: {changes: []} to remove an inherited schedule while retaining the initial network settings. Initial impairments may be omitted entirely to start unconstrained and enable them at a later deadline; these Peers also receive NET_ADMIN.

Experiment-relative schedules synchronize with the Controller before installing initial impairments. Missing start time or unsuccessful clock synchronization fails Peer startup. Each Peer then runs its own monotonic deadline timer, independent of Controller polling and subsequent clock refreshes. OS scheduling and tc execution can still make a change late; if multiple deadlines have already elapsed when the timer runs, it applies the latest due cumulative state. scope: all can delay telemetry delivery but does not suspend this local timer.

A delay-only update changes the existing netem qdisc in place. Changes to other settings rebuild KPL's qdisc tree to remove obsolete limits; this can flush queued packets and briefly interrupt shaping. Each successful application emits network_config_applied with the effective configuration, revision and lateness; a runtime tc error emits network_config_failed and fails the Peer. See monitoring for the distinction between applied configuration and measured traffic.

Timers end when their Peer stops, including cancellation and repeated-run cleanup. A naturally completed single run can retain its Peers, so their schedules continue until those Peers stop. Keep the scenario active through the intended observation window and use stop-all for a bounded experiment. The complete examples/network-schedule.yaml demonstrates both clocks, restoration, staggered joins and explicit cleanup. Deploy updated Controller, Agent and Peer images together before using this syntax.

The recommended scenario format is version 3 YAML. It retains version 2's execution controls, job tracking, and readiness barriers, and adds mesh freezing, graph formation and runtime profile schedules. The old line-oriented .kpl DSL is not parsed directly; translate its commands into phases.

For join, count is the exact number of create operations. For publish and leave, it is a maximum capped by the number of eligible candidate nodes; publish candidates must have PubSub and publishing enabled and must have joined the requested topic. repeat repeats the entire phase. On these three actions, parallel selects sequential or concurrent replica execution, while parallelism optionally caps concurrent operations. await defaults to true; setting it to false starts a tracked background job named by job, allowing later phases to run while churn or publishing continues. A wait-jobs phase can join selected jobs by name.

Version 3 schedule changes the effective profile of selected Peers and subsequent matching joins in the same run generation. group, profile, type, role, nodeIds and peerIds intersect. At least one of group, profile or an ID list is required. A profile selector can select several groups; a group selector changes only that group even if another group uses the same profile. Profile templates and saved scenario YAML remain unchanged. Explicit IDs must resolve to active Peers; an empty group/profile cohort still changes later matching joins. Readiness is checked before dispatch, including containers already admitted but still starting.

- action: schedule
  group: workers
  profile: baseline
  job: conditions
  await: false
  timeout: 2m
  schedule:
    reference: phase-start
    changes:
      - after: 10s
        set:
          gossipsub: {params: {d: 8}}
          network: {delay: 100ms}
      - after: 30s
        set:
          gossipsub: {params: {d: 6}}
          network: {delay: 10ms}

Only the listed runtime settings are supported; construction settings such as router, topics, heartbeat timers, queue sizes and scoring cannot be changed by this action. GossipSub changes require enabled GossipSub. Degree repair follows ordinary heartbeats; already frozen mesh membership stays frozen. Parameter changes do not guarantee an immediate target degree. HopWave factor/interval changes retain the existing activation setting.

For group/profile selection, validation checks each matching join and the cumulative states of this schedule, including future joins in the generation. ID selections check actual targets at runtime. Matching joins that may receive network changes are automatically created with NET_ADMIN, including when initial conditions have no impairment. A schedule network patch cannot target a Peer with an active network.schedule; GossipSub-only updates may coexist with that existing local network timer. Changes to delay alone preserve the qdisc tree; structural network changes follow the existing netem replacement behavior.

Before changing existing Peers, the Controller validates the full cohort and waits for every prepare acknowledgment. It dispatches up to 16 requests concurrently, then applies the same prepared values. New admissions are coordinated with this operation: an already admitted Peer joins the cohort; later admissions inherit the successful patches. Each Peer command is bounded separately (8 seconds at the Peer, 10 at the Agent, 12 at the Controller). HTTP/OS scheduling and shaping can make changes late; this is not simultaneous cluster-wide application. After apply begins, failures or missing acknowledgments may leave partial changes. They fail the action and Run; no automatic apply retry or distributed rollback is performed. Peers with failed applications reject further profile updates. Uncertain Agent network acknowledgments mark the current conditions pending until observed again.

stop-all/reset cancels and drains schedule jobs and clears inherited patches for the next generation. Stop, failure and normal onExit: cancel cancel pending timers. A naturally completed retained Peer keeps its last applied settings, with no remaining Controller schedule job. Retry, resume and repeated Runs begin with their original templates and new schedule state. Concurrent schedules are serialized when applying; overlapping selectors with equal deadlines do not promise a seed-determined order. Use one ordered changes list or explicit job barriers when ordering matters.

Controller schedule_change events preserve requested patches, revisions, stage, acknowledgment counts and lateness. Peer profile_config_applied records confirmed effective settings; profile_config_failed reports incomplete application. These are configuration observations, not measured latency or mesh convergence. See the internal command API and complete schedule example. Rebuild and deploy Controller, Agent and Peer images together.

Enable gossipsub.meshFreeze: true when creating each target Peer, then issue mesh-freeze after readiness and a suitable settling period. Readiness alone does not establish mesh convergence. The protocol reference defines pinned membership and the behavior of disconnects, topic changes, and continuing message traffic.

version: 3
name: fixed-mesh
profiles:
  fixed:
    type: full
    gossipsub:
      meshFreeze: true
phases:
  - action: join
    group: workers
    profile: fixed
    count: 8
    parallel: true
  - action: wait-ready
    group: workers
  - action: wait
    duration: 15s
  - action: mesh-freeze
    group: workers
    timeout: 30s
  - action: stop-all

At least one selector is required:

IDs within one list are alternatives; different fields intersect. An explicit ID must resolve inside that intersection in the current run and generation. Unknown IDs, mismatched group/type/role filters, duplicate or blank IDs, and an empty selection are errors. Stopped and failed nodes are excluded from group selection. Every selected Peer must be ready, use enabled GossipSub with meshFreeze: true, and belong to an online Agent; a failed precheck prevents dispatch to the whole selection. Join future Peers separately and freeze them with a later command; an earlier group command does not apply to future members.

The phase waits for every selected Peer to acknowledge. timeout bounds dispatch and acknowledgments for the whole selection and defaults to 30s; it must be positive. Each Agent-to-Peer request and Peer handler still has a ten-second limit, so a longer phase timeout does not extend an individual request's deadline. repeat is allowed because freezing is idempotent. await: false, parallel: true, parallelism, count, interval, lifetime, topic, and onError are unsupported for this action. Targets are dispatched with bounded concurrency. Once dispatch begins, errors or timeouts can leave some Peers frozen; successful acknowledgments do not imply simultaneous application across Peers. There is no unfreeze action.

examples/scenario-v3-mesh-freeze.yaml includes publication after freezing. Runtime HTTP payloads are in the internal API reference.

A domain is the selected set of existing Peers. Use the same intersecting group, nodeIds, peerIds, type and role selectors as mesh freeze. Each target must be ready in the current run/generation, have a connection address, enable GossipSub with meshFreeze: true, and subscribe to or relay the explicit topic. count is optional; when positive it requires exactly that many selected Peers and does not create or sample Peers. Use a preceding join and wait-ready to create the population.

For group selection without explicit IDs, scenario validation requires a preceding matching join since the last stop-all/reset. Every such join must resolve to gossipsub.meshFreeze: true after preset, profile and inline overrides; omitted or false settings fail validation with the topology phase, group and offending join name. The check respects type/role filters and includes background joins. It conservatively checks all matching joins in that reset interval, even after leave or a configured lifetime, because validation does not predict which Peers survive. ID selections (including IDs combined with a group) resolve at runtime, where every selected Peer is checked before commands are sent. topology does not enable the capability automatically.

For 500 selected Peers and an Erdős–Rényi graph:

- action: topology
  group: domain-a
  count: 500
  topic: kpl/graph
  timeout: 5m
  topology:
    model: er
    p: 0.05
    seed: 1001

The complete 500-Peer example includes creation, readiness, publication and cleanup. It needs Swarm capacity for the workers and bootstrap Peer. A separate mesh-freeze phase is unnecessary: topology installs the graph and freezes it. A previously frozen different mesh cannot be replaced; use new Peer processes for another graph.

The topology block uses these lowercase model keys and requires every parameter shown. Parameters belonging to another model are rejected.

These are simple undirected graphs: no self edges or duplicates. Connectivity is a model outcome; isolated components are retained without adding repair edges. Grid and triangular lattices have open boundaries. The geometric coordinates define graph edges and are saved with the plan; they do not configure netem latency or replace the Dashboard layout. The stochastic conventions follow the official NetworkX definitions for ER, WS, BA, RGG and Waxman. KPL implements its own generator; matching another library's seed does not promise matching edges.

Targets are ordered by KPL node ID. topology.seed overrides the run seed, including explicit zero. The same ordered population, graph parameters and seed reproduce the same graph. Generation is limited to 10,000 Peers and 1,000,000 undirected edges, checks cancellation, and fails before dispatch if a limit is exceeded. Each control request is also limited to 1 MiB.

timeout bounds target selection, graph generation, preparation and application together. When omitted, it is calculated from the selected population N as 2m + 5s × (ceil(N / 16) − 1): 2m for up to 16 Peers, 4m35s for 500, and 7m10s for 1,000. An explicit positive timeout remains a hard limit; an existing scenario with timeout: 2m keeps that limit. Remove the field to use the population-based default, or set a larger duration when the graph and network need it.

At most 16 Peer commands run concurrently, and each Peer makes at most four concurrent connection attempts. The Controller retries only transient preparation failures, within the same overall timeout and using the same generated plan. Application is not automatically retried. The request limits, retry conditions and delays are defined in the topology API.

Every target must acknowledge preparation before application begins. Each Peer then verifies and installs its exact neighbors and freezes its topic meshes in one router operation. The designated topic is replaced; other existing topic meshes retain their membership and also freeze. Existing degree, score and backoff selection do not substitute another graph. Direct peers, blacklisted/filtered neighbors and non-mesh protocols are rejected.

repeat must resolve to 1; background jobs, parallel, parallelism, pacing, lifetimes and onError are unsupported. A preparation failure installs no generated meshes, although preparation can establish transport links and ordinary GossipSub remains active. Application is atomic per Peer; network failures can leave a subset frozen, and a lost response may hide a successful application. Successful phases require every exact-plan acknowledgment. Applied plans allow identical retries, with no rollback or unfreeze.

The graph fixes the GossipSub mesh. Transport/DHT connections, direct/flood publication and lazy gossip follow their existing rules and can use paths outside it. Set connection budgets high enough for the chosen graph and other enabled protocols. The example disables worker DHT/topic discovery to reduce extra connections; it retains bootstrap and normal lazy recovery.

Saved topology_generated, per-node topology_assignment and topology_stage events preserve generator version, seed, intended adjacency and acknowledgment counts. Peer topology_applied events record local installation. Planned edges are not evidence of live links; the live topology continues to report active mesh members. Full adjacency records are excluded from the recent-event feed. See the runtime API.

Background-job behavior at the natural end of the phase list is controlled by top-level onExit. Its default, cancel, cancels remaining jobs and then waits for them to stop. onExit: drain instead waits for them to complete naturally. A naturally successful single run applies this job policy but leaves Peer containers running unless the scenario contains an explicit stop-all. A Dashboard/API batch submitted with repetitions > 1 automatically fences and cleans up Peers after every iteration, including the last, so they cannot overlap the next run. This is separate from a YAML phase's repeat field.

jobShutdownTimeout defaults to 3m. When a user or API request cancels a scenario, or when any scenario phase or background job fails, the Controller cancels outstanding jobs and waits for their termination within this bound, then asks every Agent to generation-fence and clean up Peer containers through the current generation. An explicit stop-all uses the same bounded job shutdown, resets job tracking, and fences the current run generation. The Agent records the monotonically increasing fence before stopping matching containers: a late create at generation N either committed before the fence and is included in cleanup, or is rejected because its generation is at or below the fence. After stop-all succeeds, the scenario advances to generation N+1, so later phases may create new nodes under the same run ID and may reuse job IDs.

wait-ready evaluates the complete matching cohort in the current run generation, including failed, stopping, and stopped nodes; nodes from previous generations are ignored. A failed cohort member prevents the barrier from succeeding, and a node reported as ready contributes to the ready count only while its Agent is online. Because the cohort still contains only nodes observed so far, wait-ready after an await: false join must specify either jobs: [job-id] or minCount. jobs waits for the selected producer jobs to finish before checking readiness; minCount leaves them running but prevents a partially created group from satisfying the ratio too early.

For v2 compatibility, a publish phase with no interval has a special default: with parallel: true, every operation gets a 1s phase-start offset; sequential publish uses zero delay. This rule is independent of await.

Sequential join, publish, and leave run the first operation immediately, then schedule each operation at the batch start plus the sum of preceding sampled intervals. Request/admission time consumes the interval instead of extending it. If dispatch falls behind, remaining operations execute in order without extra waiting until they catch up; arrivals are not skipped. Cancellation and errors still stop dispatch, and Agent capacity still limits admission. Parallel join and leave ignore interval. Parallel publish uses each sampled interval as that operation's independent offset from the batch start, before acquiring a concurrency slot; offsets are not cumulative and capacity can delay dispatch further.

For sequential joins, the Controller logs churn join schedule delayed when dispatch is at least one second late, at most once every ten seconds per job execution. The log includes the run, group, operation number, scheduled time, and delay. This exposes sustained capacity or API bottlenecks; the schedule cannot guarantee Docker startup or readiness times under overload. Agent heartbeats run independently of event forwarding, so a slow telemetry request does not pause lease renewal.

wait durations and readiness/job timeouts must be positive. An omitted join lifetime means no automatic leave, while an explicitly sampled 0s lifetime stops the new node immediately, matching v2.

Lifetime expiry uses the monotonic clock captured immediately after successful container creation. Waiting for Agent metadata locks or scheduling the timer goroutine consumes the sampled lifetime. An overdue deadline triggers stop as soon as the waiter can run; CPU contention and cleanup can still delay actual removal. Process cancellation releases the timer, and an already canceled process does not invoke another lifetime stop.

phases:
  - name: background churn
    action: join
    job: churn
    await: false
    parallel: false
    group: churners
    role: worker
    type: light
    count: 100
    interval: {model: exponential, mean: 250ms}
    lifetime: {model: pareto, xm: 45s, alpha: 2.5, max: 3m}

  - action: wait
    duration: 10s

  - action: wait-ready
    group: churners
    jobs: [churn]
    readyRatio: 1
    timeout: 5m

Intervals and lifetimes support these distributions. Duration-valued fields use Go duration strings, and optional min/max duration bounds clamp the result. An optional distribution may be omitted or empty. If it contains parameters, model is required; for example, lifetime: {value: 5s} is invalid and must specify model: fixed.

A non-zero scenario seed reproduces sampled delays, distribution samples, and random selection/order when the initial state, eligible candidate set, and readiness/job barriers are the same. Peer identity is deterministic per (run ID, per-node seed): the same pair yields the same Peer ID, while different run IDs intentionally yield different Peer IDs and avoid cross-run identity collisions. Thus the same scenario seed reproduces sampling and ordering across runs, but not Peer IDs when their run IDs differ; seed: 0 intentionally chooses a time-based seed. Start with examples/smoke.yaml, then see examples/mixed-workers.yaml for custom profiles, heterogeneous worker types, bounded parallel batches, and background paced jobs.

The Docker runtime isolates each Peer and supports per-node P2P egress conditions. wait-ready confirms Peer initialization and API readiness; it does not verify mesh convergence. Periodic topic discovery repairs transport candidates under churn, while GossipSub heartbeat and GRAFT processing still need time to converge. Add a settling phase when the experiment needs it. Scenario seeds reproduce application sampling and ordering under the documented conditions, but do not promise identical kernel packet impairment or network timing. Optional HopWave forwarding requires compatible builds and explicit activation.

The Agent keeps a bounded cache of acknowledged terminal history and stores older records locally; re-registration replays them in bounded pages. See Agent history retention for limits and recovery rules. Full Controller snapshots still include retained node history. Routine partial heartbeats, discovery, and dashboard SSE use active nodes and pending updates, avoiding repeated full-history processing. Set an explicit parallelism for large parallel phases; omitting it still permits the whole batch to run concurrently.

Korean translation: 한국어.