design phase 9 w7.1 dsh event tap - sinrimin/harness-nexus GitHub Wiki
Status: SHIPPED 2026-09-10 (daemon
0.11.0-p9w7.1, branchfeat/p9-w7.1-dsh-tap) โ implementation decisions and rig results in ยงPost-ship notes. Predecessor:design-phase-9-w7-native-sessions.mdยง "dsh live streaming" (the transcript-file tail, SHIPPED as daemon0.10.1-p9w7โ becomes the FALLBACK when this lands). User intent (verbatim): ๆปๆ่งๆซๆไปถไธๆฏไธไธช้ฟไน ไน่ฎก๏ผๆไปฌ่ฝไธ่ฝไนๆทปๅ ็ฑปไผผ ๆไปถ่ฟ DSH๏ผๅจๅ้็ๆถๆบๆทปๅ ่ฟๅป โ i.e. mirror the reference project's in-process plugin approach instead of tailing the transcript file. References: dsh source at~/acp-ref/dsh(checked out at tagdsh-v0.1.2-rc.1= the rig's installed version; index in~/acp-ref/README.md); the reference bridge excerpts at~/acp-ref/dsh-streaming-ref/(engine loop / chunk-rows codec / itsupdates.jsmapper โ the semantics W7 already mirrors).
W7's dsh streaming reads the transcript FILE. That works (rig-verified) but
couples us to the persistence encoding (multi-frame zstd, MIN_RUN=3 packed
rows vs verbatim events, torn-frame buffering) and costs a write-behind
window (~100โ300ms) + a 250ms poll interval per delta, plus a settle wait at
turn end. The reference project streams by subscribing dsh's in-process
session event bus (ctx.on('session/event')) from a cordis plugin โ zero
latency, uniform event shapes, no encoding concerns. An external ACP client
cannot reach that busโฆ but we don't have to be external: the daemon is the
process parent of every chat adapter, and dsh has an official spawn-time
overlay for extra plugins.
-
dsh --patch <path>exists: "extra patch-list overlay applied after the profile layer (repeatable)" (dsh --help). - Path-based plugin mounts: a patch yml of
- insert:\n - id: hnx-tap\n name: /abs/path/index.mjscomposes into--profile acp --dump-config(87 โ 88 entries). NOTE: patch entries are id-targeted โ a bare- id: <new>errors withentry "<id>" not found; NEW rows require the- insert:directive. - The plugin actually loads and runs at boot: a test plugin's
apply(ctx)executed in a one-shotdsh --profile headless --patch โฆ "reply with just the word ok"run (task completed, dispose marker written). - The bus subscription works: the same plugin counted 22
session/eventdeliveries during that one short task โ the streaming raw material, in-process, zero latency. - Constraint: dsh's stdio belongs to the ACP JSON-RPC protocol โ the tap must NOT write to stdout/stderr protocol content; it talks to the daemon over its own localhost socket.
hnx daemon (chat session start, target deepseek)
โโ TapListener: node:net server on 127.0.0.1:0 (ephemeral port)
โ env: HNX_TAP_PORT=<port> HNX_TAP_TOKEN=<one-time random>
โโ spawn: dsh --profile acp --patch <tapDir>/patch.yml โ assets shipped inside the hnx npm package
โ
โโ dsh process
โโ hnx-tap plugin (zero-dep .mjs, insert-mounted)
ctx.on('session/event') โโโถ TCP JSON-lines โโโถ TapListener
โ filter by acpSessionId
โผ
createDshLiveMapper (SAME semantics as the tail)
โ
wire committed chunks suppressed (unchanged)
Priority chain (purely additive โ worst case equals today's behavior):
- Tap handshake within 3s of spawn โ tap is the streaming source; do NOT attach a TranscriptTail at all.
- No handshake (old dsh,
--patchunsupported, plugin load failure,HN_DISABLE_DSH_TAP=1A/B switch) โ today'sensureTailfile tail, unchanged. - Neither โ committed-only (pre-W7-addendum behavior).
- Files:
src/daemon/dsh-tap/index.mjs(hand-written ESM, zero deps, never imports from@harness-nexus/*) andsrc/daemon/dsh-tap/patch.yml(- insert:entry pointing atindex.mjsby ABSOLUTE path is impossible in a static file โ sopatch.ymlis a TEMPLATE and the daemon renders it next to the plugin at boot, or simpler: the daemon writes a tiny per-boot patch yml into the hnx config dir (~/.hnx/dsh-tap.patch.yml) referencing the package-absolute plugin path. Decide during implementation; the spike proved the shape). - Plugin body (~60 lines):
export const name = 'harness-nexus-tap';export function apply(ctx)readsHNX_TAP_PORT/HNX_TAP_TOKENfrom env,node:net.connect(127.0.0.1, port), first line{"type":"hello","token":โฆ,"pid":โฆ}, then one JSON line per bus event:{"type":"event","sessionId":<session.id>,"event":<session event verbatim>}. Forward ALL sessions' events (the DAEMON filters) โ the plugin stays dumb. Reconnect with a small backoff while the process lives;ctx.effectdisposal closes the socket. No config surface. - Bus event shape = the verbatim session events W7 already maps
(
{type:'assistant/chunk', seq, time, data:{turn, step, chunk}}etc.) โ persistence packing (text-chunksrows) NEVER appears on the bus, so the tap path exercises only the simple half of the existing mapper.
The rig overlay procedure copies ONLY packages/*/dist, and npm files
ships dist โ so the tap assets must land in dist/daemon/dsh-tap/ at
build time (a node -e "fs.cpSync(...)" step appended to the cli build
script; tsc won't emit .mjs/.yml verbatim). Runtime resolution from
dist/daemon/chat.js: new URL('./dsh-tap/', import.meta.url) โ works in
the repo dev tree, the npm install, AND the container overlay.
- New
packages/cli/src/daemon/dsh-tap-listener.ts: net server, token check on the hello line (mismatch โ destroy socket), buffered JSON-line parsing,onEvent(sessionId, event)callback,close(). -
chat.ts(deepseek arm only): create listener BEFOREAcpAgentConnection. start(the child needs the env); append['--patch', <rendered yml>]to the resolved dsh argv (the fixture agent must tolerate the extra argv โ verify; it likely ignores argv entirely); race handshake vs 3s timeout; on handshake: feed events through the SAMEcreateDshLiveMapperinstance (one per session), filtersessionId === acpSessionId, suppression keys off the tap exactly as it keys off the tail today; on timeout: fall through toensureTail(the existing replay-eligibility logic applies unchanged โ a tap session never attached the tail, sowireTextEmittedsemantics carry over). - Turn-end ordering: keep the settle pattern but source it from the tap โ
after the wire settles, wait (โค600ms) for the tap's
turn/endfor this turn before emittingturn_result(typically instant on the bus). - Teardown: close listener + socket with the session (add to
teardown).
- The wire suppression rule, the mapper semantics, the history ring, resync,
resume flows,
TranscriptTailitself (stays as fallback + still parses resume history), the server/web โ zero server or web changes, no new daemon capability, no schema changes. Daemon version bump only.
-
Subagent sessions: bus events carry the emitting session's id โ the
daemon filters to
acpSessionId, so subagent noise (parentSession/ delegationDepth sessions in the same process) never renders. -
dsh upgraded, cordis API broke: plugin throws/never connects โ 3s
timeout โ file tail. Feature-detect, never fatal. The tap depends only on
apply(ctx)/ctx.on/ctx.effectโ the same surface the reference bridge uses in production. - Two chat channels on one machine: one dsh process per channel (C5 model), each with its own listener/port/token โ no cross-talk.
-
Fixture agent (tests):
HN_ACP_COMMAND_DEEPSEEKoverrides the command; appended--patchargv must be ignored by the fixture (it reads JSON-RPC on stdin and ignores argv โ confirm; if not, strip unknown flags there). -
Do NOT write the patch into
~/.dshpersistent config โ spawn-time--patchonly. The homecordis.patch.ymlis the user's surface (W3 manages its marked region) and applies to every profile; our overlay must stay scoped to the processes we spawn.
-
Unit (plugin): import
index.mjsin a test, callapply(fakeCtx)with a stubctx(on/effectrecorded); assert handshake + event forwarding against a local net server; assert token mismatch โ server destroys; assert dispose closes. -
Unit (listener + integration via fixture): fixture agent gains a
"tap-speaker" mode (or the daemon's listener is tested directly): a
spawned node process connects per
HNX_TAP_PORTenv and replays canned bus events โ assert deltas stream, wire committed chunks suppressed, turn/end settles turn_result, fallback engages when the tap never connects (existing tail tests already cover path 2). -
Rig E2E (A/B): re-run the archived streaming probe
(
~/acp-ref/rig-stream-probe.mjsโ same three-turn assertions: count / tool / post-resume) with the tap live; then withHN_DISABLE_DSH_TAP=1to prove the tail fallback still passes; compare first-delta latency tap vs tail (expect the write-behind window + poll interval to disappear); leak-scandsh --profile acpafter the run.
- Tap assets + build copy step + runtime path resolution (verify
dist/ daemon/dsh-tap/survives npm pack and the container overlay). -
dsh-tap-listener.ts+chat.tsspawn composition + fallback chain + settle-from-tap. - Fixture tolerance for
--patchargv + the two unit-test layers. - Local gates:
pnpm -r typecheck, cli tests, build. - Rig: overlay per
docs/dev/test-rig.md(git-ignored, on this machine), daemon restart, probe A/B, leak scan. NEVER push without the user's ask. - Docs: this file โ Status SHIPPED + post-ship notes; AGENTS.md W7 section
addendum;
docs/dev/test-rig.mdentry; DAEMON_VERSION bump (0.11.0-p9w7.1); branchfeat/p9-w7.1-dsh-tapโgit merge --no-ff.
- Upstreaming native streaming to
@deepseek-ai/dsh-acp(itspackages/acp/acp/src/updates.tsself-describes as "derived from committed DSH session events" โ the exact gap; a separate issue, for which the reference mapper is the proof of concept). - Taps for claude-code/codex (their adapters already stream token-level).
- Removing
TranscriptTail(it remains the fallback and the resume-history parser). - Any persistent mutation of the user's
~/.dsh(spawn-time overlay only).
Landed exactly as ยงArchitecture draws it. Where the design left a decision open, implementation resolved:
-
No static
patch.ymlasset ships. The daemon renders~/.hnx/dsh-tap.patch.ymlitself at every deepseek session start (idempotent overwrite;dsh-tap-listener.tswriteTapPatch) โ one file serves every channel because the per-channel port/token ride the child env, not the patch. Onlyindex.mjsrides the package (build-copied intodist/daemon/dsh-tap/). -
The 3s handshake races the SPAWN, not the ready.
chat.tsarms the listener beforeAcpAgentConnection.startandPromise.alls the hello against initialize: a healthy plugin loads during dsh's composition, so the race resolves at zero added latency; establishment failure closes the listener in the existing kill-catch (leak-hygiene intact). A SLOW-but- healthy boot (>3s to hello) still degrades to the tail โ worst case is W7 by construction. -
Tap death mid-session โ committed-only for that session (
tapDead):ensureTailrefuses to attach afterwards. A fresh tail mapper cannot know whichturn:steps the tap already streamed, so byte-0 replay or commit-dedup would double-render; the tail-corruption path makes the same trade. -
The plugin bounds its reconnects (~6s of exponential backoff, 10
attempts) and stops permanently on the listener's
{"type":"reject"}line (wrong token = the port belongs to someone else โ never spam it). - Tests: the fixture gained
FIXTURE_TAP_SPEAKER=1(speaks the plugin protocol with canned bus rows โ tests must pinFIXTURE_SESSION_ID, since the daemon filters tap events by the acp session id) andFIXTURE_ARGV_FILE(asserts the--patchcomposition and thatHN_DISABLE_DSH_TAP=1omits it entirely).
Rig results (2026-09-10, daemon 0.11.0-p9w7.1, standard dist overlay โ
no server/web rebuild):
-
A (tap, default):
rig-stream-probe.mjsPROBE PASS ร2 (24 assertions). Turn 1 (count 1..25): 107 deltas, median inter-delta gap 1ms, first delta 810ms โ per-LLM-chunk granularity. -
B (tail,
docker exec -d -e HN_DISABLE_DSH_TAP=1 โฆ hnx daemon): PROBE PASS (24/0 on rerun; the first run hit 23/1 โturn1: progressive deltas (>=4)saw 3 batches, the known tail batch-count variance the W7 addendum documents; assertions there must stay span/slot-based). Turn 1: 3โ4 deltas, median gap 212msโ3.5s, first delta ~3s. - The write-behind window + 250ms poll interval are gone on the tap path,
exactly as ยงProblem predicted. Leak scan (
/proc/*/cmdlinefordsh --profile acp) after 8 channel openings across both runs: zero.
dsh โฅ0.1.3 moved per-token deltas OFF the session event bus (see the W7 #46 addendum: the bus keeps settlement rows only). The tap as shipped would handshake fine but stream nothing token-grained. Tap v2 subscribes the SECOND bus and forwards frames; the shipped plugin asset is version- agnostic (both subscriptions coexist โ an older dsh simply never emits
agent/assistant-stream).
Plugin (daemon/dsh-tap/index.mjs): a second ctx.on('agent/assistant- stream', ({agent, frame}) => โฆ) writes {"type":"frame","sessionId": agent.session.id,"frame":frame} lines next to the event lines. Same
constraints as before (never touch stdio, swallow errors, bounded reconnects;
ctx.effect disposes both subscriptions).
Listener (dsh-tap-listener.ts): TapListenerEvents.onFrame? โ a new
frame line type, same sessionId filtering contract.
Daemon (chat.ts): the per-session tap binds THREE callbacks now โ
onEvent (turn/end arms the settle signal; settlement/tool rows through
mapper.event()), onFrame (deltas through mapper.frame()), onLoss.
The frame arm's start record supplies the turn:step that the settlement
arm's dedup keys on (chunk frames carry only the attemptId โ the mapper
keeps attemptId โ turn:step internally, so chat.ts stays stateless about it).
Fixture: FIXTURE_TAP_SPEAKER=1 now speaks frames (start + text-delta
chunks, incl. a SUBAGENT-session frame the daemon must filter), then the
streamed-step commit + bus turn/end as before โ the W7.1 integration test's
assertions (deltas stream, wire commits suppressed, settle gated on
turn/end, usage rides the commit) all hold unchanged under the new protocol.
#48 post-ship fix (2026-10-14): the hello window rides the START; the prewarm pool never caches tap-less boots
Rig incident: a user turn rendered as ONE burst at turn end. Post-mortem found two coupled defects, both v4-amplified:
- v4 session logs have no intra-turn rows โ every durable row of a turn (assistant/message, step/end, turn/end) flushes in the final write-behind batch. The transcript tail therefore SETTLES, it does not stream: on dsh 0.2 the tap is the only streaming source a deepseek channel has. A tap-less channel is not "degraded to W7 streaming" anymore โ it has none, for its whole life.
-
The hello deadline was a fixed 3s from ARM (pre-spawn). Slow cold boots
(small machines, concurrent spawns, first boot after a config push) can
legitimately outlive it โ standalone rig measurement shows ~1.2s hello when
idle, but the incident's pooled spawn missed the window. On a miss the
PREWARM POOL published the tap-less boot anyway (
{started, tapArmed: null}), and the web's open adopted it (~310ms ready = pooled) โ locking the channel into no-streaming.
Fix (both ends):
-
armTapexposeswaitHello(deadlineAt)instead of a pre-baked promise; both the channel open and the pool spawn now decidetapLiveat start-completion + grace (HN_TAP_HELLO_GRACE_MS, default 2500). Healthy taps say hello long before initialize resolves, so the grace is paid only when the plugin is truly absent (the open is that much later, committed-only). The retired constant wasTAP_HANDSHAKE_MS = 3000. - A pool boot whose tap never handshakes is killed and dropped โ never published. The next open spawns fresh with a new arm; the kill leaves its ledger entry for the W11 audit like any deliberate kill.
Fixture: FIXTURE_TAP_DELAY_MS (delayed dial โ the slow-boot shape) and
FIXTURE_EXIT_FILE (exit marker; the fixture exits cleanly on SIGTERM so
'exit' handlers run). Tests: the pool drop (marker + prewarmReady false +
recoverable fresh open) and the slow-hello win (dial at +4s, grace 6000 โ
past the retired 3s window, inside start+grace).