RECITATION SPEC - SyberLabs/RISE GitHub Wiki
Written 2026-07-28, revised 2026-07-29 against a frame-by-frame reading of the reference reel and a measured TTS probe.
A spoken presentation of the authored source through a completely admitted static voice pack. Recitation may provide timing evidence to Progressive Reveal, but does not own text arrival. Reveal and speech are independently selectable.
A reading mode and a renderable contract, not an export implementation. Interactive playback and deterministic MP4 rendering now consume the same compiled timing authority. Recitation specifies the voice and reveal evidence that both paths receive; the render pipeline owns frames, encoding, and the artifact rather than screen-recording the browser.
The reel is two acts, and only the second is R.I.S.E.'s business.
Act one (0:00β8:25) β captions over filmed footage, in word
mode: OKAY Β· IF Β· YOU Β· HAD Β· 15 Β· MINUTES, each word
replacing the last at 150β350ms. This is word-chunked RSVP, which
R.I.S.E. already does. The footage is not ours to generate.
Act two (8:25β25:30) β a hard cut to black, then a montage in phrase mode, where each phrase reveals itself rather than appearing:
I β I AM β I AM THE β I AM THE ALPHA.
That is one atom revealing progressively, not four atoms replacing
each other. Measured from the frames: 750ms reveal, then 1000ms held
β a 43% reveal share, about 50ms per character. Then imagery, then
THE FIRST. held 1400ms, then more imagery, then AND at 450ms, then
THE END split across a composition.
So the two acts differ in chunk mode, and the "typewriter" is a reveal within a phrase atom.
| element | state |
|---|---|
| Word / phrase chunking | Both modes exist and are selectable. |
| Held sacred imagery | The Chapel pins works; Gallery and full-frame present them. |
| Speech |
AudioEngine.speak() is complete at engine.js:2064 with onStart/onEnd hooks β and has never had a caller. |
| Ducking | The exact ramp pattern exists for the Shuttle. Not wired to voice. |
| Per-atom colour | Living Text tints by valence. Driven by the wrong signal for this β see Β§3. |
| Keystroke sound |
layerGains.typing with a sampler and configured volume, built for something else. |
| Progressive Reveal | Implemented as an independent revealMode; voice may provide its word clock. |
An atom's duration is computed from word count at the session WPM, and
the reader is promised that long to read it. A progressive reveal breaks
that promise silently β the phrase is not readable until the reveal
completes, so the reader gets less time than the duration implies.
The reveal is independent of speech, and behaves differently with it.
- Without speech β the reveal completes within a share of the atom's duration; the remainder is reading time at rest. The temporal contract is preserved exactly.
- With speech β the reveal follows the voice. Words appear as they are spoken, and the atom lasts as long as the utterance. The WPM setting stops governing, which is correct: when a reading is spoken, the voice is the clock.
That second case transfers authority away from a setting the reader chose, so it must be visible in the UI rather than surprising.
reveal = min(REVEAL_MAX_MS, duration Γ REVEAL_SHARE).
The reel measures 43% and ~50ms per character. Proposed:
REVEAL_SHARE = 0.4, REVEAL_MAX_MS = 800. A short atom is not
swallowed by its own animation; a long one does not crawl.
Atoms under ~400ms already bypass the fade path in displayAtom and
appear whole. Revealing a three-word atom that lives 300ms is a
strobe, not a reveal β those keep appearing whole.
The reel reveals I β I AM β I AM THE, which is word by word.
Character-by-character is a different effect and a worse one here: a
reader tracking half-words is decoding, not reading. Words also give the
speech case a natural unit to sync against.
prefers-reduced-motion disables the reveal entirely β text appears
whole. Not "reveals faster": off.
speak() is wired to the player's atom advance, and its onEnd
advances the reading rather than the computed duration.
Runtime decision, 2026-07-29: browser neural inference is retired. The browser-provisioning material below is retained as the measurement record that led to the decision, not as the current implementation.
Kokoro-82M and kokoro-js are local authoring tools. The pack builder runs under Node and emits content-addressed WAV assets plus a bundled manifest:
npm run build:voice-pack -- --input scripts/voice-packs/heart-beta.mjs
Production receives only same-origin audio. A reader's browser downloads no model, creates no ONNX worker, requests no Hugging Face resource, and uses no speech API or account. There is no recurring synthesis or server cost; deployment pays only ordinary static-file storage and bandwidth.
The browser normalizes each speakable phrase, hashes it, and verifies the exact normalized text at that manifest key. It requires complete session coverage before fetching audio, so a partial pack can never stutter in and out. It fetches and decodes eight initial phrase assets, then maintains a twelve-asset lead with concurrent static requests.
The builder validates finite samples, audible energy, and safe peak before
writing Kokoro's own WAV output. Duration and speech-onset metadata are
computed once and stored in the manifest. The first admitted pack is Heart
(af_heart) for the complete Library Meditations selection in Phrase
mode. The UI lists only installed packs and Spoken mode selects Phrase
chunking to keep compilation and asset identity aligned.
Owner decision, 2026-10-03: today's poem is recited by ElevenLabs v4, rendered once at build time on the owner's paid plan.
The rule above still holds: a reader's browser runs no model and calls no
speech service. What changes is who renders the audio before it is
committed. scripts/build-poem-recitation.mjs (npm run poem:recite) reads
ELEVENLABS_API_KEY from the author's own environment only; the key is
never in the repository, CI or the browser.
-
One performance per poem. Each poem is rendered whole through
/with-timestamps, so prosody runs across lines. Per-line synthesis would read every line as a sentence. -
Cut at the reader's lines.
src/audio/poem-alignment.jsmaps the character timing onto the reader's own atoms (phrase mode with verse lines: one atom per line), letter by letter. A performance that skips or adds a word is refused, and that poem stays silent until it is re-taken. Each line is cut at the quietest 10 ms between lines, with 4 ms fades, and carries one speech onset per word. -
One small pack per poem. Clips are AAC
.m4a, 64 kbps mono, named by content hash. Each poem's pack is an ordinaryrise.recitation-voice-pack.v1manifest under/audio/recitation/<voice>/<sha16>.json;src/audio/poem-recitation.jsonnames one pack per poem. The session carries it asrecitation.pack(validated against that path shape) andVoicefetches it. Anything other than that manifest (a 404, the app shell, a wrong schema) leaves the reading silent, exactly as an uncovered session is. -
Drift is caught, not heard. The index is honoured only when its
sourceRevisionmatches the released edition, andpoem-recitation.integration.test.jsfails if the chunker changes the lines a pack covers. Responses are cached under.cache/poem-recitation/, so a re-cut costs nothing.
window.speechSynthesis is a formant synthesiser and no setting makes
it sound human. Replaced by Kokoro-82M via kokoro-js, both
Apache-2.0, running fully client-side β see TTS-RESEARCH.
There is no assumed browser reliability baseline. The earlier Node
benchmark selected q4f16/WASM because it measured at RTF 0.34β0.38, but
Node uses onnxruntime-node, not the browser's onnxruntime-web.
Deployed Chrome and Edge later produced non-finite q4f16 samples for
both Heart and Fenrir, while q8/WASM produced healthy audio too slowly
to sustain the Chamber. Both conclusions are measured.
The full five-voice harness confirmed this on 2026-07-29 under controlled frame pressure: single-thread q8/WASM ran at RTF 2.60β3.03; q4f16/WASM ran at RTF 1.86β2.09 and returned sample-zero non-finite output for Bella's passage and two Fenrir lengths. A sparse q8 Fenrir transient peaked at 1.082 without exceeding the clipping-ratio threshold; that is recorded as a warning, not conflated with q4f16's numeric corruption.
fp32/WebGPU then measured at steady-state RTF 0.11β0.41 after shader warm-up, but seven of fifteen combinations produced finite numeric explosions, including a peak of 2.7 Γ 10Β²βΆ. Because validation occurs inside the worker on Kokoro's raw samples, this rejects WebGPU before encoding, transfer, or playback enter the path.
A provider is admitted only after the deployed-browser harness verifies every offered voice and representative fragment, sentence, and passage. Every run must contain finite, healthy samples and p95 generation RTF must be at most 0.75, leaving 25% replenishment headroom for normal variance and the Chamber's visual workload. βThe model loadedβ is not an audio-health or throughput result.
The harness lives at /tts-harness.html. It creates a fresh worker and
ONNX session per provider, measures q8/WASM, q4f16/WASM, optional
uint8/WASM, and fp32/WebGPU when available, retains Kokoro's own WAV for
audition, and exports the complete browser/device evidence as JSON.
Until a provider passes on the target browser/device class, Recitation
must treat speech as unavailable rather than choosing a backend by
feature detection alone.
Once a provider is qualified: cold start before entering the Chamber, then buffer by audio duration, not phrase count. Generation remains serial because ONNX permits one run per session, and queued work remains reader-aware so text the reader has passed is discarded. Once the Chamber is visible, generation never holds the reader. A missing or invalid segment degrades to the authored silent timer and cannot poison the rest of the startup lead.
If the buffer runs low, stop speaking and let the reading continue
silently. Never stall the reading to wait for audio. Authored
[PAUSE]/empty atoms do not count as underruns. A bounded state-transition
diagnostic records starvation and recovery without flooding the console.
Reverent degradation applies to speech exactly as it does to imagery.
The signal analysis below now runs in the local pack builder. Its onset result travels in the manifest; the deployed browser performs no Kokoro generation or waveform analysis.
kokoro-js exposes no per-word timestamps β generate() returns
{audio, sampling_rate} and nothing else. But it returns raw
Float32Array samples, and 20ms RMS windows find the silences between
words directly: a 7-word phrase yielded 6 gaps, onsets at
0.30/0.80/1.12/1.62/1.92s.
The reveal is anchored to real speech onsets, not a blind interpolation. The first detected word remains aligned to its measured onset. The remaining onset span is compressed to 70%, so written words lead comprehension slightly and a run of short words cannot leave the visual reveal trailing the voice.
RISE does not re-encode those samples. Normal playback copies them directly into Web Audio; media fallback uses Kokoro's own IEEE Float32 WAV Blob. A parallel PCM16 encoder is both unnecessary and a format drift risk.
The reel colours emphasised words cyan against white: 15,
MINUTES, BEAUTIFUL, AMAZING, LORD, JESUS, CHRIST. That is
semantic, not decorative β it marks what the sentence is for.
Emphasis is authored, exactly as phrase boundaries are. The Vault's
sequences already carry | marks placed by a human; emphasis is the
same kind of notation and belongs in the same layer. Content authors;
the runtime follows.
Living Text is not the right source. It computes valence, and
sentiment is not emphasis β BEAUTIFUL scores high and 15 does not,
yet the reel emphasises both. Deriving emphasis from sentiment would
mark the wrong words confidently.
The two coexist: Living Text tints the atom by mood; emphasis marks particular words within it. One is climate, the other is stress.
Emphasis also affects timing. DO holds 950ms where its neighbours
hold 200ms β the question landing. An emphasised word earns dwell, which
the temporal system can already express.
Checked: an asterisk notation passes through the chunker untouched β
I would tell you how *beautiful* and *amazing*... survives phrase
chunking as one atom with the marks intact. So emphasis can be authored
in the text and parsed at the display layer, the same seam | uses.
The marks are stripped before display and before speech, exactly as
| and [PAUSE] already are.
One real consequence. displayAtom currently sets
atomDisplay.textContent = atom.content at two sites. Text content
cannot carry per-word colour, and it cannot be revealed word by word.
Both the reveal and emphasis require the atom to be built from
per-word spans instead.
That is a change to the hottest path in the reading loop, so it must:
- keep the existing fast path for atoms under ~400ms, which appear whole and need no spans at all;
- escape the text, since building HTML from content is a new injection
surface where
textContentwas inherently safe; - and leave the non-recitation path byte-identical, so an ordinary reading is not paying for a feature it is not using.
Serif type on black with a blurred halo. Glow is a text-shadow
derived from the atom's Living Text colour, so the two never disagree β
one signal, two expressions.
Strongest on the word being revealed, settling as the phrase completes. If that proves fussy, a whole-atom glow that brightens during the reveal and settles after is an acceptable simplification.
The static WAV is decoded into an AudioBufferSourceNode on the shared
graph. Media fallback plays the fetched WAV Blob.
Full-frame Rhythmic presence is a semantic pause. The completed phrase finishes before the visual opportunity; a next phrase prepared behind the opaque overlay keeps every word pending, does not begin speaking, and does not start its reading clock until the visual has fully resolved. Its measured word reveal, WAV, and reading clock then resume together without emitting or laying out the atom a second time. Concealed DOM preparation is never a text or audio entrance.
Music and narration always coexist at their authored levels. Recitation does not automatically duck in Full-frame, Attractor, Focal, Behind-stream, Gallery, or any other presentation.
The engine retains a dormant, tested musical-layer ramp primitive for a future authored dramatic structure such as quote β synth accent β continued quote. That behavior must be a precise content cue with an explicit span, not an ambient consequence of enabling speech or selecting a visual mode.
Verified: the graph is named layer gains each connected to
masterGain. Static decoded samples enter through a mono
AudioBufferSourceNode connected directly to masterGain, outside the
named musical layer gains. A future cue may therefore shape musical layers
without attenuating narration; ui and typing remain outside that
primitive. The authored WAV Blob remains the compatibility fallback when no
shared Web Audio graph exists.
The existing three-way control (Full-frame / Behind-stream / Gallery) describes how imagery is presented. This describes how text is presented. They are orthogonal β a recitation can run under a Gallery β so folding them into one control would be a category error of the kind the Library's shelves just went through.
Recitation and Reveal both belong beside WPM and chunking in the temporal orbit, but remain separate controls. The four legal combinations are instant or progressive text, each with voice off or on.
-
The temporal contract holds when the voice is off. The reveal
borrows from
duration, never extends it. - Reverent degradation. No model, slow device, error mid-utterance β the reading continues silently. It never stalls waiting for audio.
- Reduced motion disables the reveal. Off, not faster.
- Emphasis is authored. The runtime never guesses which words matter.
- One signal, two expressions. Glow derives from the Living Text colour rather than computing its own.
- The Chapel is unaffected. Sacred readings have chant beds and liturgical pacing. Synthetic speech over scripture is a decision nobody has made, and this spec does not make it.
Published from docs/vision/RECITATION-SPEC.md. Edits made here are overwritten on the next push to main.