RECITATION SPEC - SyberLabs/RISE GitHub Wiki

Recitation β€” spoken reading and acoustic balance

Written 2026-07-28, revised 2026-07-29 against a frame-by-frame reading of the reference reel and a measured TTS probe.


0. What this is

A spoken presentation of the authored source through a completely admitted static voice pack. Recitation may provide timing evidence to Progressive Reveal, but does not own text arrival. Reveal and speech are independently selectable.

A reading mode and a renderable contract, not an export implementation. Interactive playback and deterministic MP4 rendering now consume the same compiled timing authority. Recitation specifies the voice and reveal evidence that both paths receive; the render pipeline owns frames, encoding, and the artifact rather than screen-recording the browser.

What the reference reel actually does

The reel is two acts, and only the second is R.I.S.E.'s business.

Act one (0:00–8:25) β€” captions over filmed footage, in word mode: OKAY Β· IF Β· YOU Β· HAD Β· 15 Β· MINUTES, each word replacing the last at 150–350ms. This is word-chunked RSVP, which R.I.S.E. already does. The footage is not ours to generate.

Act two (8:25–25:30) β€” a hard cut to black, then a montage in phrase mode, where each phrase reveals itself rather than appearing:

I   β†’   I AM   β†’   I AM THE   β†’   I AM THE ALPHA.

That is one atom revealing progressively, not four atoms replacing each other. Measured from the frames: 750ms reveal, then 1000ms held β€” a 43% reveal share, about 50ms per character. Then imagery, then THE FIRST. held 1400ms, then more imagery, then AND at 450ms, then THE END split across a composition.

So the two acts differ in chunk mode, and the "typewriter" is a reveal within a phrase atom.

What already exists

element state
Word / phrase chunking Both modes exist and are selectable.
Held sacred imagery The Chapel pins works; Gallery and full-frame present them.
Speech AudioEngine.speak() is complete at engine.js:2064 with onStart/onEnd hooks β€” and has never had a caller.
Ducking The exact ramp pattern exists for the Shuttle. Not wired to voice.
Per-atom colour Living Text tints by valence. Driven by the wrong signal for this β€” see Β§3.
Keystroke sound layerGains.typing with a sampler and configured volume, built for something else.
Progressive Reveal Implemented as an independent revealMode; voice may provide its word clock.

1. The reveal

The problem it creates

An atom's duration is computed from word count at the session WPM, and the reader is promised that long to read it. A progressive reveal breaks that promise silently β€” the phrase is not readable until the reveal completes, so the reader gets less time than the duration implies.

The rule

The reveal is independent of speech, and behaves differently with it.

  • Without speech β€” the reveal completes within a share of the atom's duration; the remainder is reading time at rest. The temporal contract is preserved exactly.
  • With speech β€” the reveal follows the voice. Words appear as they are spoken, and the atom lasts as long as the utterance. The WPM setting stops governing, which is correct: when a reading is spoken, the voice is the clock.

That second case transfers authority away from a setting the reader chose, so it must be visible in the UI rather than surprising.

Budget, without speech

reveal = min(REVEAL_MAX_MS, duration Γ— REVEAL_SHARE).

The reel measures 43% and ~50ms per character. Proposed: REVEAL_SHARE = 0.4, REVEAL_MAX_MS = 800. A short atom is not swallowed by its own animation; a long one does not crawl.

Atoms under ~400ms already bypass the fade path in displayAtom and appear whole. Revealing a three-word atom that lives 300ms is a strobe, not a reveal β€” those keep appearing whole.

Granularity: word, not character

The reel reveals I β†’ I AM β†’ I AM THE, which is word by word. Character-by-character is a different effect and a worse one here: a reader tracking half-words is decoding, not reading. Words also give the speech case a natural unit to sync against.

Reduced motion

prefers-reduced-motion disables the reveal entirely β€” text appears whole. Not "reveals faster": off.


2. Speech

speak() is wired to the player's atom advance, and its onEnd advances the reading rather than the computed duration.

Runtime decision, 2026-07-29: browser neural inference is retired. The browser-provisioning material below is retained as the measurement record that led to the decision, not as the current implementation.

Runtime architecture: static voice packs

Kokoro-82M and kokoro-js are local authoring tools. The pack builder runs under Node and emits content-addressed WAV assets plus a bundled manifest:

npm run build:voice-pack -- --input scripts/voice-packs/heart-beta.mjs

Production receives only same-origin audio. A reader's browser downloads no model, creates no ONNX worker, requests no Hugging Face resource, and uses no speech API or account. There is no recurring synthesis or server cost; deployment pays only ordinary static-file storage and bandwidth.

The browser normalizes each speakable phrase, hashes it, and verifies the exact normalized text at that manifest key. It requires complete session coverage before fetching audio, so a partial pack can never stutter in and out. It fetches and decodes eight initial phrase assets, then maintains a twelve-asset lead with concurrent static requests.

The builder validates finite samples, audible energy, and safe peak before writing Kokoro's own WAV output. Duration and speech-onset metadata are computed once and stored in the manifest. The first admitted pack is Heart (af_heart) for the complete Library Meditations selection in Phrase mode. The UI lists only installed packs and Spoken mode selects Phrase chunking to keep compilation and asset identity aligned.

Vendor-rendered packs (build time)

Owner decision, 2026-10-03: today's poem is recited by ElevenLabs v4, rendered once at build time on the owner's paid plan.

The rule above still holds: a reader's browser runs no model and calls no speech service. What changes is who renders the audio before it is committed. scripts/build-poem-recitation.mjs (npm run poem:recite) reads ELEVENLABS_API_KEY from the author's own environment only; the key is never in the repository, CI or the browser.

  • One performance per poem. Each poem is rendered whole through /with-timestamps, so prosody runs across lines. Per-line synthesis would read every line as a sentence.
  • Cut at the reader's lines. src/audio/poem-alignment.js maps the character timing onto the reader's own atoms (phrase mode with verse lines: one atom per line), letter by letter. A performance that skips or adds a word is refused, and that poem stays silent until it is re-taken. Each line is cut at the quietest 10 ms between lines, with 4 ms fades, and carries one speech onset per word.
  • One small pack per poem. Clips are AAC .m4a, 64 kbps mono, named by content hash. Each poem's pack is an ordinary rise.recitation-voice-pack.v1 manifest under /audio/recitation/<voice>/<sha16>.json; src/audio/poem-recitation.json names one pack per poem. The session carries it as recitation.pack (validated against that path shape) and Voice fetches it. Anything other than that manifest (a 404, the app shell, a wrong schema) leaves the reading silent, exactly as an uncovered session is.
  • Drift is caught, not heard. The index is honoured only when its sourceRevision matches the released edition, and poem-recitation.integration.test.js fails if the chunker changes the lines a pack covers. Responses are cached under .cache/poem-recitation/, so a re-cut costs nothing.

Historical browser-inference record (retired)

window.speechSynthesis is a formant synthesiser and no setting makes it sound human. Replaced by Kokoro-82M via kokoro-js, both Apache-2.0, running fully client-side β€” see TTS-RESEARCH.

There is no assumed browser reliability baseline. The earlier Node benchmark selected q4f16/WASM because it measured at RTF 0.34–0.38, but Node uses onnxruntime-node, not the browser's onnxruntime-web. Deployed Chrome and Edge later produced non-finite q4f16 samples for both Heart and Fenrir, while q8/WASM produced healthy audio too slowly to sustain the Chamber. Both conclusions are measured.

The full five-voice harness confirmed this on 2026-07-29 under controlled frame pressure: single-thread q8/WASM ran at RTF 2.60–3.03; q4f16/WASM ran at RTF 1.86–2.09 and returned sample-zero non-finite output for Bella's passage and two Fenrir lengths. A sparse q8 Fenrir transient peaked at 1.082 without exceeding the clipping-ratio threshold; that is recorded as a warning, not conflated with q4f16's numeric corruption.

fp32/WebGPU then measured at steady-state RTF 0.11–0.41 after shader warm-up, but seven of fifteen combinations produced finite numeric explosions, including a peak of 2.7 Γ— 10²⁢. Because validation occurs inside the worker on Kokoro's raw samples, this rejects WebGPU before encoding, transfer, or playback enter the path.

Continuous narration requires provider qualification

A provider is admitted only after the deployed-browser harness verifies every offered voice and representative fragment, sentence, and passage. Every run must contain finite, healthy samples and p95 generation RTF must be at most 0.75, leaving 25% replenishment headroom for normal variance and the Chamber's visual workload. β€œThe model loaded” is not an audio-health or throughput result.

The harness lives at /tts-harness.html. It creates a fresh worker and ONNX session per provider, measures q8/WASM, q4f16/WASM, optional uint8/WASM, and fp32/WebGPU when available, retains Kokoro's own WAV for audition, and exports the complete browser/device evidence as JSON. Until a provider passes on the target browser/device class, Recitation must treat speech as unavailable rather than choosing a backend by feature detection alone.

Once a provider is qualified: cold start before entering the Chamber, then buffer by audio duration, not phrase count. Generation remains serial because ONNX permits one run per session, and queued work remains reader-aware so text the reader has passed is discarded. Once the Chamber is visible, generation never holds the reader. A missing or invalid segment degrades to the authored silent timer and cannot poison the rest of the startup lead.

Degradation

If the buffer runs low, stop speaking and let the reading continue silently. Never stall the reading to wait for audio. Authored [PAUSE]/empty atoms do not count as underruns. A bounded state-transition diagnostic records starvation and recovery without flooding the console. Reverent degradation applies to speech exactly as it does to imagery.

Timing for the reveal

The signal analysis below now runs in the local pack builder. Its onset result travels in the manifest; the deployed browser performs no Kokoro generation or waveform analysis.

kokoro-js exposes no per-word timestamps β€” generate() returns {audio, sampling_rate} and nothing else. But it returns raw Float32Array samples, and 20ms RMS windows find the silences between words directly: a 7-word phrase yielded 6 gaps, onsets at 0.30/0.80/1.12/1.62/1.92s.

The reveal is anchored to real speech onsets, not a blind interpolation. The first detected word remains aligned to its measured onset. The remaining onset span is compressed to 70%, so written words lead comprehension slightly and a run of short words cannot leave the visual reveal trailing the voice.

RISE does not re-encode those samples. Normal playback copies them directly into Web Audio; media fallback uses Kokoro's own IEEE Float32 WAV Blob. A parallel PCM16 encoder is both unnecessary and a format drift risk.


3. Emphasis

The reel colours emphasised words cyan against white: 15, MINUTES, BEAUTIFUL, AMAZING, LORD, JESUS, CHRIST. That is semantic, not decorative β€” it marks what the sentence is for.

Emphasis is authored, exactly as phrase boundaries are. The Vault's sequences already carry | marks placed by a human; emphasis is the same kind of notation and belongs in the same layer. Content authors; the runtime follows.

Living Text is not the right source. It computes valence, and sentiment is not emphasis β€” BEAUTIFUL scores high and 15 does not, yet the reel emphasises both. Deriving emphasis from sentiment would mark the wrong words confidently.

The two coexist: Living Text tints the atom by mood; emphasis marks particular words within it. One is climate, the other is stress.

Emphasis also affects timing. DO holds 950ms where its neighbours hold 200ms β€” the question landing. An emphasised word earns dwell, which the temporal system can already express.


Notation and the display path

Checked: an asterisk notation passes through the chunker untouched β€” I would tell you how *beautiful* and *amazing*... survives phrase chunking as one atom with the marks intact. So emphasis can be authored in the text and parsed at the display layer, the same seam | uses. The marks are stripped before display and before speech, exactly as | and [PAUSE] already are.

One real consequence. displayAtom currently sets atomDisplay.textContent = atom.content at two sites. Text content cannot carry per-word colour, and it cannot be revealed word by word. Both the reveal and emphasis require the atom to be built from per-word spans instead.

That is a change to the hottest path in the reading loop, so it must:

  • keep the existing fast path for atoms under ~400ms, which appear whole and need no spans at all;
  • escape the text, since building HTML from content is a new injection surface where textContent was inherently safe;
  • and leave the non-recitation path byte-identical, so an ordinary reading is not paying for a feature it is not using.

4. Glow

Serif type on black with a blurred halo. Glow is a text-shadow derived from the atom's Living Text colour, so the two never disagree β€” one signal, two expressions.

Strongest on the word being revealed, settling as the phrase completes. If that proves fussy, a whole-atom glow that brightens during the reveal and settles after is an acceptable simplification.


5. Interlocution and acoustic mixing

The static WAV is decoded into an AudioBufferSourceNode on the shared graph. Media fallback plays the fetched WAV Blob.

Full-frame Rhythmic presence is a semantic pause. The completed phrase finishes before the visual opportunity; a next phrase prepared behind the opaque overlay keeps every word pending, does not begin speaking, and does not start its reading clock until the visual has fully resolved. Its measured word reveal, WAV, and reading clock then resume together without emitting or laying out the atom a second time. Concealed DOM preparation is never a text or audio entrance.

Music and narration always coexist at their authored levels. Recitation does not automatically duck in Full-frame, Attractor, Focal, Behind-stream, Gallery, or any other presentation.

The engine retains a dormant, tested musical-layer ramp primitive for a future authored dramatic structure such as quote β†’ synth accent β†’ continued quote. That behavior must be a precise content cue with an explicit span, not an ambient consequence of enabling speech or selecting a visual mode.

Verified: the graph is named layer gains each connected to masterGain. Static decoded samples enter through a mono AudioBufferSourceNode connected directly to masterGain, outside the named musical layer gains. A future cue may therefore shape musical layers without attenuating narration; ui and typing remain outside that primitive. The authored WAV Blob remains the compatibility fallback when no shared Web Audio graph exists.


6. Where it lives

The existing three-way control (Full-frame / Behind-stream / Gallery) describes how imagery is presented. This describes how text is presented. They are orthogonal β€” a recitation can run under a Gallery β€” so folding them into one control would be a category error of the kind the Library's shelves just went through.

Recitation and Reveal both belong beside WPM and chunking in the temporal orbit, but remain separate controls. The four legal combinations are instant or progressive text, each with voice off or on.


7. Invariants

  • The temporal contract holds when the voice is off. The reveal borrows from duration, never extends it.
  • Reverent degradation. No model, slow device, error mid-utterance β€” the reading continues silently. It never stalls waiting for audio.
  • Reduced motion disables the reveal. Off, not faster.
  • Emphasis is authored. The runtime never guesses which words matter.
  • One signal, two expressions. Glow derives from the Living Text colour rather than computing its own.
  • The Chapel is unaffected. Sacred readings have chant beds and liturgical pacing. Synthetic speech over scripture is a decision nobody has made, and this spec does not make it.

Published from docs/vision/RECITATION-SPEC.md. Edits made here are overwritten on the next push to main.

⚠️ **GitHub.com Fallback** ⚠️