Design review 2026-08-02/03 (Kaetemi). Resolves the W2 blocker in tutorial roadmap — "SNP hash-list streaming is not integrated with the Emscripten build and isn't pipelined". Status: design agreed, nothing implemented. To be revised further.

.snp is the right shape and the wrong interface.

The format is a manifest with no payload — SNPK + version + {Name, SHA1, Size, LastModified} per entry (streamed_package.cpp:38-57) — with bytes out-of-band at content-addressed paths /aa/bb/rest (streamed_package.cpp:59-65). That is what an HTTP cache wants.

What blocks it is one line of contract: IStreamedPackageProvider::getFile returns a filesystem path (i_streamed_package_provider.h:36) and CIFile then fopens it (file.cpp:227-235).

Why the shape beats .bnp on the web:

• .bnp is all-or-nothing — CBigFile::getFile hands back FILE* + offset (big_file.h:79), so the whole archive must exist in the FS.
• Content-addressed immutable URLs mean the browser HTTP cache does the job CHttpPackageProvider re-implements by hand (temp files, local hash tree, exists-check at http_package_provider.cpp:62).
• Dedup and partial updates fall out for free.
• CFile::getFileSize answers from the manifest without downloading (path.cpp:2077-2092).

The honest alternative — .bnp + HTTP Range against the trailing index (big_file.h:96-99) — is feasible but loses dedup, invalidates the whole URL on any repack, and depends on browser range-caching behavior. .snp uses the cache instead of fighting it.

1. Path-returning provider. Forces materializing every streamed file into the FS. Fix: buffer/IStream-returning provider entry point plus a memory branch in CIFile alongside the snp branch at file.cpp:218. CIFile already has a memory-cache path (file.h:184,198) — that is the cheap seam.
2. Blocking on the main thread. Samples link -s ASYNCIFY, no -pthread (gui/CMakeLists.txt:75 and all of samples/3d/*); p_thread.cpp:305 concedes single-threaded mode has no scheduling. Survivable via emscripten_fetch under Asyncify, but serial and stack-unwinding per asset. The manifest is the unwritten fix — every hash and size is known up front, so prefetch can fan out in parallel before the main loop, leaving Asyncify as the miss handler.
3. curl. CHttpPackageProvider is synchronous curl-easy (http_package_provider.cpp:75-110); the emscripten build gets only a stub curl (CMakeLists.txt:397-410). A browser provider is a new class regardless, and belongs outside nelweb.
4. LZMA is the wrong codec for the web — see §4.
5. Manifest weight — see §4c.
6. Fatal parse failure. loadPackage calls nlerror on serial exception (streamed_package_manager.cpp:57), which aborts the wasm module. A 404'd or truncated manifest is a normal transient on the web; must become a warning + false.

• path.cpp:1414-1417 — CPath::addSearchStreamedPackage delegates to _FileContainer.addSearchBigFile. The static entry point does the wrong thing entirely. Only the auto-detect route (path.cpp:1182-1185, calling the member) works today.
• streamed_package_manager.cpp:44-47, 62-66 — m_Entries stores const CEntry* into the package's std::vector. Reloading a package name destroys the old vector; entries that vanished leave dangling pointers. The failure path at :58 erases while stale pointers may persist.
• streamed_package_manager.cpp:75-76 — list() dereferences find() without an end() check.
• streamed_package_manager.cpp:109-111 — unreachable return true.
• snp_make/main.cpp:226-233 — update scan is O(entries × files); ~10⁹ string compares at Ryzom corpus scale. :354-355 — -l output has no newlines.

The delta engine ports: CXDeltaPatch::apply is internal C++ (login_patch.cpp:2929-2950 → login_xdelta.cpp:789) needing only zlib gzFile (login_xdelta.h:155-183). Everything around it does not.

• Patching requires owning the storage. You cannot write into the browser HTTP cache — a patch produces new bytes for the same URL, which is what an HTTP cache is built to refuse. So you need OPFS/IDBFS you manage, which means hand-rolling the download manager and giving up the free cache.
• Browser storage is not durable. Absent a granted navigator.storage.persist(), the UA may evict the origin. A patcher's premise is "the local copy is a known version N"; the browser reserves the right to delete that silently.
• Cost model is hostile. CPatchThread::processFile applies chains sequentially, whole-file, materializing .tmp__j per step (login_patch.cpp:2797-2900). N patches on a 300 MB bnp = N full read+write cycles and 2× file size free quota at each step. Native already flinches — see the bail-out heuristic at :2712.
• mtime trap. Version identification is (filesize, mtime ±2s) (:1784-1795) with SHA1 rescan as fallback, and applyDate (:1674) writes it back. Emscripten FS backends do not round-trip mtime reliably; if it does not survive, every startup falls through to hashing the entire data set.
• Port surface: five thread classes (CCheckThread, CPatchThread, CScanDataThread, CDownloadThread, CInstallThread); curl (:1419); createBatchFile/executeBatchFile/reboot (:725, :1004, :1079) which are meaningless in a browser.

Content addressing dissolves the problem rather than solving it. A changed file is a new hash, therefore a new immutable URL; unchanged files keep their URL and cache entry. Delta update at file granularity, free, with no patch generation, no version chains, no _.ref files, no batch files. patch_gen's server-side apparatus stops existing.

Bnp+xdelta only wins on intra-file deltas — a large file that changes in small ways. Rare; address case by case if it ever bites.

Not a new architecture — the native one with the installer replaced by a fetch. The client already has installed bnps plus streamed snp, and CPath already treats both as one namespace.

1. Embed — bootstrap only (fallback font, loading shaders). --embed-file, as samples/3d/font/CMakeLists.txt:13 already does. Kilobytes.
2. BNP bundle tier — required/small files. One request for N files; CBigFile works as-is in MEMFS (plain nlfopen/seek/read with thread-local handle caching, big_file.cpp:150,713). Engine work: fetch into MEMFS, then CBigFile::add.
3. SNP streamed tier — large, individually addressable, genuinely conditional assets. Contents lazy; manifests fetched at boot.

Rule that keeps the split honest: anything with size goes to snp even if required (put it on the prefetch warm list); anything small goes to bnp even if optional. snp is "fetched individually", not "optional" — see the measured cost in §4b.

Why bnp rather than --preload-file (what samples do today, gui/CMakeLists.txt:66-71): a preload .data blob is welded to one binary, so changing one texture invalidates the blob and ties it to that build. A versioned bnp URL lets binary and assets version and cache independently.

The seam is free: CPath inserts both tiers into the same lookup map with the same pack@name convention and extension remapping (path.cpp:1445-1470), so files move between tiers with zero call-site change.

Every bnp and snp lives at an immutable versioned or content-addressed URL, Cache-Control: immutable. Consequences:

• Patching evaporates (§2).
• No quota, no OPFS, no IndexedDB, no eviction handling. Everything rides the HTTP cache. The moment anything needs to write persistent storage, all of that comes back.
• Old and new clients coexist; a stale tab keeps working against its pinned set.
• ryzom_%05d.idx plus all of patch_gen's apparatus collapse into one ~1 KB immutable JSON per client version.

Exactly one mutable object in the system. Preferably push it up to the HTML wrapper itself (version baked in at deploy), so even the index is immutable.

Consider content-addressing the bnps too — <sha1>.bnp — so immutability is intrinsic rather than conventional and bnps dedupe across client versions.

The JS wrapper reads the version manifest and fetches the initial data before the wasm module starts.

• Asyncify never touches the boot path — fetch is natively async.
• Parallelism free via Promise.all.
• The .wasm, the bnps, and the snp manifests download as one parallel wave (today --preload-file serializes them).
• Real HTML/CSS loading screen with byte-accurate progress, rendering before wasm exists.

Mechanism is emscripten's own: Module.preRun plus addRunDependency/removeRunDependency. Use FS.createDataFile(..., canOwn=true) to transfer the fetched ArrayBuffer rather than copying it — without it you hold both the fetch result and the MEMFS copy.

Note: MEMFS stores contents as JS-side typed arrays, not wasm linear memory. Bundles do not consume the wasm32 4 GB address space or interact with ALLOW_MEMORY_GROWTH realloc cost. Only bytes actually read land in the wasm heap. Resident for the session either way.

Keep the wrapper dumb — it fetches a list it was handed. All policy (category selection, the main_exedll_* platform filtering at login_patch.cpp:1250-1313, required-vs-optional) resolves at manifest-generation time, or you get asset policy in two languages.

Do not parse ryzom_%05d.idx in JS — it is the persistent_data format (DECLARE_PERSISTENCE_METHODS in bnp_patch.h). Emit a small JSON sidecar from patch_gen; .idx stays canonical for native.

Handoff back to C++: write the same manifest JSON into MEMFS and let startup code read it, then addSearchBigFile / addSearchStreamedPackage per entry. (Fix path.cpp:1414-1417 first, or call the container method.)

Skip a Service Worker initially — the HTTP cache does this job. It only earns its complexity for genuine offline play.

Streaming during play. CAsyncFileManager is a CTaskManager — a background thread (async_file_manager.h:29-34). Single-threaded wasm has none, so zone loads, async textures, and snp misses land on the main thread and block the frame. This is the next hard problem, and it decides whether a browser client is playable or merely bootable. Answer is either -pthread (SharedArrayBuffer, COOP/COEP headers on the host) or reworking the async loaders around Asyncify with a per-frame time budget.

Cold start is worse than a native install — first visit pays the full bnp tier before anything renders. That is the number to watch as the design ages.

457 files, 46,202,880 bytes. 225 dds, 96 zonel, 52 ig, 47 tga, 12 shape, 9 anim, 3 swt, 3 ps, 1 skel, 1 rbank, 1 gr, 1 bank, 1 farbank, fonts.

Whole set, bundled (tar):

Per directory, bundled, zstd-19:

Window size (zstd-19, whole tar) — plateaus at 8 MB; transport-cap worries are noise at this scale:

Tiles (225 DDS, ~29 KB average), held-out test (trained on odd files, measured on even):

Bundling beats a dictionary by 2.7× — the dictionary is capped at 64 KB while a bundle gets the full window.

Dictionary gain by file size, held-out: 19.6% on 29 KB tiles, 3.4% on 190 KB zones. Dictionaries only earn their keep on small files that must be individually addressable — i.e. the snp tier and nowhere else.

Per-file vs bundled, zstd-19 no dict: shapes 9.99× vs 10.83×, anims 3.47× vs 4.19×, zones 3.61× vs 4.27×.

Consequence: every file moved from bnp to snp costs roughly 2.7× in bytes for that file. Bundle aggressively; compress the bundle.

Emit both .zst (zstd-19) and .br (brotli-11) for every bundle; let Accept-Encoding negotiate. zstd is the primary artifact — the native client consumes the same .zst, so one codec, one library in patch_gen, one artifact serving both targets. Brotli exists purely as a compatibility fallback nothing else depends on.

Content-Encoding: zstd support as of 2026-08: Chrome/Edge 123+, Firefox 126+, Opera 109+, Safari 26+ (partial) / 26.3+ (full). Gaps: Samsung Internet, Safari ≤25 (i.e. anyone on an older iOS). That tail is exactly what content negotiation covers.

Practicalities:

• Precompress at build time, max quality. On-the-fly CDN brotli is ~q4; offline q11 is free at serve time.
• Header footgun. Serving foo.bnp.br without Content-Encoding: br delivers garbage. Prefer brotli_static-style server config over encoding-in-filename.
• Skip already-compressed payloads — ogg/jpg gain ~nothing. Measure per bundle; a bnp that is 90% ogg is not worth precompressing.
• GitHub Pages cannot do this. It does not let you set Content-Encoding or supply precompressed variants, and typically compresses only text MIME types — so a .bnp as application/octet-stream likely ships uncompressed (46 MB instead of 12.6 MB). Verify with curl -H 'Accept-Encoding: zstd, br' -I against a binary on the existing Pages site. Assume assets must go to cdn.ryzom.dev, which pulls CORS into round one.

Hashes are incompressible, so brotli does almost nothing for a .snp. The lever is the encoding. Current per-entry cost (streamed_package.cpp:48-57): 1 (per-entry serialVersion) + 4+name + 4 (length prefix on a fixed-width CHashKey) + 20 hash + 4 Size + 4 LastModified ≈ 37 + name.

v2 target ≈ 18 + name:

• Version the container, not each record.
• Drop the length prefix on the fixed-width hash.
• Drop LastModified from the client-facing manifest (keep it in a build-side sidecar for snp_make's incremental rebuild).
• Varint the sizes.
• Truncate the hash to 128 bits — ample for collision avoidance over ~10⁵ files when transport is HTTPS and the hash does no security work.

At 30k entries: ~2 MB → ~1 MB, on the critical path before anything renders.

46.2 MB / 457 files — small enough to run a full build→publish→deploy cycle in minutes while the pipeline is still wrong. That is the whole reason; it has no unique correctness property (see §5d).

• Source: ~/snowballs_source (105 MB — max, maps, tilebank, ligo, veget_set, sfx, fonts).
• Reference built data: ~/snowballs_reference/data (46 MB), from snowballs-data-20030801.zip.
• Corpus already green (T1/T2 111, zone 96, anim 9; LIGHT tier gated) — lighting config recovered, see zone lighter config reversal and Snowballs.

Do not tier Snowballs. 12.6 MB compressed is one fetch. Ship one bnp. Generate a second, split manifest as a CI fixture (zones into .snp) so the snp path stays exercised — label it a fixture in code, or someone will "fix" the shipping config to match it.

snowballs2/CMakeLists.txt:5 requires NeL 3d misc net pacs sound. NeL networking is TCP — no browser equivalent without a WebSocket bridge.

Good news: network.cpp already guards every call site with if (!isOnline()) return; (:274,283,303,319,393). Phase one is an offline single-player build — landscape, character, camera, snowball physics, no services. Multiplayer via a WS bridge is a separate later project.

Sound is probably fine (emscripten ships OpenAL and vorbis ports); verify the NeL OpenAL driver builds, and stub it for phase one if it fights — SBCLIENT_DEV_SOUND is already a toggle (snowballs_config.h:54).

Agreed scope 2026-08-03: no dedicated asset server yet. Snowballs data build is a separate job in the existing GitHub CI, source assets pulled from a static CDN archive (the ryzomcore_graphics-rev5.7z pattern already at emscripten-samples.yml:54-74), and the emscripten publish job consumes the built data.

All tooling is Linux-native and unconditionally available: WITH_NEL_TOOLS + WITH_3D + WITH_PIPELINE_NATIVE_OLE turns on every pipeline_max_export_* plus zone_lighter_2003 (nel/tools/3d/CMakeLists.txt:1-49). No WIN32 gate, no Wine, no libgsf.

Preserve asset_revision as an explicit computed value even though in round one it is just a GH Actions cache key:

asset_revision = hash(source archive id + exporter binary hashes + lighter cfg + build config)

Emit it as a file in the artifact and a field in the manifest. Phase 3 (dedicated server) then becomes a backend swap, not a redesign. The key must include the exporter binaries, or an exporter fix silently serves stale data — hashing the built binaries is cruder than tracking a dependency subtree but is self-maintaining and cannot be wrong.

Risks:

• zone_lighter is the schedule unknown — 96 zones with 8-ring dependencies on a 4-core hosted runner. Measure before wiring anything else; if a cold build is 40 minutes the plan depends on cache reliability.
• Cache eviction means cold builds happen — GH Actions caches evict after 7 days unused, 10 GB repo cap. Cold-path time is the number that matters.
• Isolate the failure — separate job, workflow_dispatch-able alone; a broken asset build must not take down the existing sample deploys.

Byte-matching reference is not a publish gate — formats diverge (recompression, re-containering, possible mobile texture variants), so it would be disabled on the first real change. Exporter-output equality against reference is meaningful and already gated upstream by the pipeline_max ctest battery.

Round-one gates:

• Completeness — every reference named by a .shape/.ig/.zonel resolves inside the bundle set. Catches the actual common failure (missing file, extension remap that did not fire).
• Load — headless parse of every asset through NeL's serializers. Catches truncation and bad container assembly.

Later:

• Render smoke, revision-to-revision — headless SwiftShader, fixed camera positions, perceptual threshold against the previous published revision, not against 2003.
• Size tripwires on bundle deltas.

Reference data keeps one narrow role: a one-time visual baseline for the first published revision, judged by eye. After that it is purely a development aid for the exporters.

1. Asset build + publish (round one: in CI). Ends with a content-addressed bnp + manifest.
2. Emscripten offline Snowballs client booting from a hardcoded manifest URL via the preRun wrapper. Ends with a playable page.
3. CI join — a resolve step replaces the hardcoded revision; deploy writes it into the wrapper. Where rev5 stops being a literal.
4. SNP fixture build, proving the streaming path, ready to carry to Ryzom.

Each phase independently useful and revertable.

Deferred past round one. When it lands:

• Storage: btrfs. Snapshot per asset_revision, compress=zstd:3, checksums on a long-lived corpus (silent corruption in a 2003 source archive would surface years later and be unattributable), send/receive to ship revisions. chattr +C the build scratch directory — repeated rewriting is exactly what fragments CoW filesystems. Alternative: XFS (reflink=1) for scratch on a separate volume.
• Reflink, not dedup. The blob store is content-addressed, so duplicates are known by construction — materialize revision trees by reflinking from the store. No scanner needed. (Skip ZFS dedup=on; you would be paying a scanner to rediscover what the naming scheme already encodes.)
• No filesystem supports pre-trained zstd dictionaries. zram does (algorithm_params, algo=zstd level=8 dict=/path) but is RAM-backed. EROFS's Z_EROFS_ZSTD_MAX_DICT_SIZE is almost certainly window size, not a trained dictionary. If dictionary-class gains are wanted: order similar files adjacently in a SquashFS/EROFS zstd image so the window catches cross-file redundancy, or use the zstd seekable format (contrib/seekable_format) — same shape as the snp tier with the archive doing the dictionary's job.
• Orchestration options, best first: self-hosted runner on the asset server (plain needs:, outbound-HTTPS only, so NAT is a non-issue — but fork PRs must never reach it); repository_dispatch (server pushes); commit status / check run; custom deployment protection rules (verify availability on the github-pages environment, emscripten-samples.yml:404-413); polling last.
• The exporters are an input. They live in this repo, so code pushes must trigger asset revalidation — not circular, just ordered: assets depend on tools-at-commit-X, client depends on assets.

• Measure zone_lighter_2003 wall time on Snowballs (blocks §5c sizing).
• curl -I GitHub Pages for Content-Encoding on octet-stream (decides §4c hosting).
• Verify NeL OpenAL driver builds under emscripten.
• Verify emscripten WASMFS OPFS backend requires -pthread (only matters if persistent storage ever returns).
• Fix the §1b bugs — independent of any of this.

• Tutorial roadmap — W1/W2/I9 entries this unblocks
• Streamed Package Provider
• Creating and Using Big Files
• Zone lighter config reversal