Codec design - NexusTomorrow/server GitHub Wiki

How the server will read and write every client message: one declaration per message, a source generator that writes its reader and writer, and a public table of the client's own layout facts that every declaration is checked against. It's build step 2 on the Roadmap. The shape is agreed in the server core design; the details on this page are still being settled, and the page moves to published when the plan for the step is agreed. Framing and transport belong to the core skeleton, not here.

Every message is declared once as a C# type whose fields carry their exact bit widths, and a source generator writes its reader, writer and size from that declaration, for both directions. A field without a width doesn't compile. A table of facts about the client's own layouts, written as our own numbers and public in the repository, checks every declaration in CI.

The message models we started from were written by hand, a Read method and a Write method per type, and the width of most fields was a default rather than a decision.

• Defaults hid the width. Every reader and writer primitive took an optional width defaulting to the C# type's size, and enums defaulted to 64 bits. Of about 3,600 primitive calls in the model code, 2,175 left the width out, and no explicit width on the write side was over 32 bits. So every 64-bit field on the write side relied on a default, which is how enums reached 64 bits without anyone noticing.
• The client disagreed. A first check against the client's own layouts found 70 models wrong: 50 fixed layouts and 20 empty paths, mostly enums at the default 64-bit width. The full survey once every model was declared, under The facts check, found 116.
• Read and write drifted apart. 55 types both read and write, and 5 of them disagree with themselves: one reads a field as a 16-bit packed float and writes it as a 32-bit float, another reads 14 bits and writes 32.

A field's width is a fact about the wire. Writing it down once, where the compiler can see it, removes the whole class of bug.

0x009A is a client-to-server message of four fixed fields, 81 bits in all. Nobody has checked yet what it's for, so it carries neutral names:

[WireMessage(GameMessageOpcode.Client009A, Direction.ClientToServer)]
public sealed partial class Client009A
{
    [UInt(32)] public uint Unknown0 { get; private set; }
    [UInt(16)] public ushort Unknown1 { get; private set; }
    [UInt(32)] public uint Unknown2 { get; private set; }
    [UInt(1)] public bool Unknown3 { get; private set; }
}

Wire order is declaration order. The width and the signedness are always explicit, never inferred from the C# type, because a float can travel as an integer and a narrow signed field can be read as unsigned. Once the message audit has established what the message and its fields mean, it takes the client's names, a summary on the type and on each field, and a [Verified] attribute saying how far it's been checked.

On a collection, the scalar attribute describes the element: [Array(CountBits = 3), UInt(18)] List<uint>. A member whose type is a [WireStruct] needs no attribute. A layout may derive from another [WireStruct], and a message from another message; either way, the base's fields come first. A message can also sit inside another layout, as a member or a list element, since the client nests whole message bodies in lists (0x0260 carries a list of 0x025F bodies), unless it holds an aligned blob, which needs the byte boundary only a message's start guarantees.

The bit reader and writer, the attributes, the layout types and the layout tree live in their own project, NexusTomorrow.Network.Codec. It's the first project marked as hot path, so the build bans LINQ, reflection, boxing conversions, streams and blocking calls in it (the code standard has the list). The reader and writer are ref structs over spans, least significant bit first as the client reads and writes, and neither has a default width.

For each declared type: a reader, a writer, a size in bits, a constant for fixed layouts, the widths on its empty path, and its full layout tree, so the facts check can compare declarations without running anything. The tree is WireLayout.TreeText, and WireLayout.Tree parses it on first use. [WireLayoutData] carries it as well, so a struct declared in another assembly nests into its holder's tree, and a test holds every declared message's tree to the empty path the facts check already proves. One registry per assembly, WireMessages, holds every message's layout and a factory per direction keyed by opcode; it replaces the old reflection walk that found message types at start-up. The write path allocates nothing: the size picks a pooled buffer, and the writer fills a span. A test measures a generated Write and requires zero allocated bytes, since the build's banned-API check can't see generated code.

A union's selector comes from its arm. The writer takes the selector value from the arm's [UnionCase] and sets the selector property to match, so the two can't disagree. In the old code the selector was a separate property set by the caller. On read, an unknown selector throws; it never stops reading quietly.

The reader picks the arm by the selector's value on the wire. A [UInt(n)] selector carries its cases' unsigned bits even on a signed type, so case 200 is 200 on the wire and a signed enum's -1 under [UInt(8)] is 255; an [SInt(n)] selector compares its cases as signed numbers. One selector serves exactly one union.

An aligned blob's alignment counts as a width on the empty path, since the writer emits it: five bits followed by an aligned 32-bit length is [5, 3, 32], 40 bits, which is what the client's facts say for that shape.

The generator refuses to emit code for a type that breaks a rule, and the build fails:

A type with any error gets no generated code at all, and neither does a type that nests or derives from it. So a bad declaration fails at the declaration, never somewhere in generated code.

The generator places each diagnostic on its declaration's syntax tree, so .editorconfig severities and #pragma reach it exactly as they reach the compiler's own: a warning such as NTC003 can be suppressed for one file, with its reason. No project suppresses either warning today.

Every layout, a declaration's or the client's, can be written as one tree in one line of text. The codec reads and writes that text with WireTree.Parse and WireTree.Format, as WireNodes.

A tree is a sequence of nodes in wire order. Structs dissolve into it: a nested struct's or a base type's nodes sit inline in the sequence that holds them, a fixed array repeats its element inline, a vector is one field per component, and a pad is fields of at most 64 bits. Only arrays and unions nest, because only they hold a sequence of their own.

Nodes are separated by one space. An index counts nodes, not bits, in the sequence the array or union sits in, and the node it names must be a field that comes earlier.

A disagreement is named by its opcode and side, then a path to its node: [4] is node 4 of the message's sequence, .element enters an array's element, and .arm 0x02 enters a union's arm for selector 2. So in the layout shaped like 0x01C3 above, [4].element[3].arm 0x02[0] is the first node of arm 2 of the union inside the fifth node's element.

One JSON Lines row per wire opcode, 1,119 rows: 366 client to server, 718 server to client and 35 both ways. A row holds the direction, and for each side whether the layout is fixed or computed, its total bits on the empty path, its field widths in order, and where its computed parts (strings, arrays, unions, blobs) sit:

{"op":407,"dir":"c2s","c2s":{"kind":"fixed","bits":64,"widths":[64]}}

The table is our own numbers, kept beside the codec and public, so public CI checks it with no client data. It's regenerated for each client build outside the repository. Every row holds for both the 64-bit and the 32-bit client of build 16042: 1,112 rows are identical between them, none differs, and 3 rows are still empty for both.

What it catches: wrong widths, missing or extra fields, wrong enum widths, opcodes swapped between different layouts, and fields swapped in order when their widths differ. What it can't see: swaps of equal width, signedness, and meaning. Generated round-trip tests cover every declared message on top of it.

The facts check is a pinned test, and with the old code deleted it's the proof the declarations stand on. Every declared message is compared with the client facts for its opcode, and each disagreement is pinned by opcode and side with its reason. The test fails on a disagreement that isn't pinned, and on a pinned one that no longer occurs, so the list only shrinks when a fix lands; a run with one pin moved to a message that agrees failed with exactly those two complaints. When the test was first pinned, at eff7f77 (run 38723274664), 811 of the 927 declared messages agreed with the client facts, and 194 facts rows had no declared message yet. The other 116 messages held 117 disagreements, pinned as 116 entries because one message has two problems on the same side:

The corrections brought it down. At 9a3a2c7 (run 38727448508), 844 messages agreed and 84 disagreements were pinned; at 07ccf71 (run 38728379667), 873 agree and 55 remain, and deleting the old code at c5b0774 (run 38847142696) changed neither number. A few stay pinned on purpose, because their widths alone can't decide the right declaration. The message audit's full-layout check takes the rest over.

The old code had shapes a plain list of fields can't say. This is how each is declared:

• A union family is a wire interface, each arm with its [UnionCase].
• A family whose arms write the shared fields after their own, which inheritance can't express, declares the shared fields in each arm, after its own, in wire order. Where the selector is written per element, the arms sit in a holder that keeps it.
• The byte-aligned envelopes are blobs whose 32-bit length counts itself; the packed one aligns after its first 5 bits.
• The network guid is a 128-bit struct of four fields.
• Byte buffers are byte arrays, fixed where their size is fixed, counted where it isn't.
• Column-by-column records are parallel lists sharing one count, and fields the old code wrote inline as a group are structs of their own.
• Fixed-count lists are arrays, padded to their count by their initializer.
• Values the old code packed in its body are declared fields.
• A list with no count before it gets the count the client reads there.
• A member the old code wrote wider than its type, chiefly an enum at the old 64-bit default, was declared at its type's width followed by a pad making up the rest, so no bits changed. The facts check then corrects each width to the client's.

Stage 2 replaced the old hand-written models with declarations, and proved nothing changed on the wire before anything was fixed. The converter and the proof were deleted at c5b0774 once that was done. This records what they did, because the declarations' first proof rests on it.

The converter read the old models with Roslyn's semantic model, statement by statement, asking for the overload the compiler chose so that a default width became an explicit one. It kept the old wire behaviour exactly, wrong widths included. It converted 1,089 of 1,254 types on its first pass and 1,165 once the types waiting on another followed; the rest were declared by hand, one union family at a time. The estimates from counting idioms beforehand had been 93% and 98%.

The proof paired every declared type with its old twin in a frozen copy of the old models. Each pair was filled from an empty instance and from at least 32 seeded random fills, enough to reach every arm of the widest union. Both writers had to put the same bytes and the same number of bits on the wire, and both readers had to read them back to the same values. A review closed three critical ways it could pass a wrong declaration and four lesser ones, and its own tests planted six kinds of mistake and caught each. At 0f25afd (run 38715582093), 1,227 of 1,281 declared types were proven equal to the old code, 3 of them against its Read only, and 28 more through the types that hold them. The 26 left were all defects in the old code: floats read as integers, reads that always threw, one list's count sent for another's, missing counts, and a few miscounts and misorders.

Then the fixes began, one commit per cause the facts check reported. The declaration follows the client, so every correction moved a type out of agreement with the old code: at 07ccf71, 1,155 were still equal to it, 27 through their holders, and 99 excluded, before the copy was deleted.

The codec is built in three stages, each ending green in CI:

1. The machinery: the bit primitives and the hot-path switch, the declaration language, the generator, and the facts table with its check, proven on hand-written test messages.
2. The conversion: the converter, the proof against a copy of the old models, and the declarations replacing the old ones.
3. The fixes: one commit per cause the facts check reports, the opcode list's errata, and deleting the old code.

Stages 1 and 2 are built. Stage 3 stripped the old Read and Write methods from the declarations (1,302 of them, with the 35 members and types only they used, both dispatch factories, the injected constructors and the reflection that registered message types, which the generated registries replace), settled the opcode list and the swapped fields, brought the facts check down to 55 pinned disagreements, and deleted the old code. It's paused there for the message audit on the Roadmap, which takes over what's left.

Retired at c5b0774: GamePacketReader, GamePacketWriter, IReadable, IWritable and MessageAttribute; the old packet layer (the client and server packet types, the fragment buffer, the bit array and the packet exception); the converter, the frozen copy of the old models and the proof against them. Nothing in the tree reads or writes a message except the generated codec. Framing comes back with the core skeleton's transport, built on the codec with a caller and tests, and writing the frame length at its true 32 bits where the old code cast it to 16. Network.World and Network.Auth came under the analysers and nullable reference types at 9c93c58, which leaves no project with either off.

These come from the approved server core design:

• One declaration per message, every field with an explicit width; a field without one is a build error.
• A source generator writes read, write and size for both directions, so the test harness's headless client uses the same declarations as the server.
• Four computed shapes on top of fixed fields: strings, counted arrays, unions picked by a type field, and byte-aligned blobs.
• A public client-facts table checks every declaration in CI, and generated round trips cover every message.

Settled in the codec's first tasks:

• A value or count that doesn't fit its field throws a CodecException, and so does a read past the end of a message's body. The old writer silently dropped the high bits, and the old reader returned 0xFF past the end; both turned a bug into a wrong value on the wire. A signed field is range-checked the same way, against the range its width can hold.
• [SInt(n)] is two's complement in n bits, sign-extended on read; [UInt(n)] is unsigned. The conversion kept each old field's bytes; which fields the client really treats as signed is checked field by field in the message audit.
• A union with no arm for selector 0 marks its layout incomplete, and the facts check reports that opcode as a failure, since its empty path can't be known from the declaration.
• "Every message declared" meant every message the old code modelled. 181 wire opcodes had no name, and the check across the whole registry counts and reports them without failing on them. The message audit widens it to every message the client registers, declaring those 181 under neutral names.

Settled in stage 3, against the client:

• The opcode list's errata. A layout the old code filed under 0x776E, a typo, moves to 0x076E, and one filed under 0x0511, which isn't a client opcode, moves to 0x0513. 0x07DF is declared once. The layouts under 0x0851 and 0x0852 swap to the opcodes the client uses for them. 0x0635 is declared in both directions, and 0x03B3, which the old code declared both ways, client to server only, as the client handles them. The opcode list has since been rebuilt as exactly the client's 1,119 wire opcodes, under neutral names.
• The two swapped field pairs, in 0x0094 and 0x0198, take the client's order.

• Nothing from the declaration design itself. What's left is the facts check's pinned list, which the message audit takes over as it checks every message's full layout, purpose and fields against the client.

• Which client fields are signed, settled message by message at the audit's R3.
• Element layouts beyond the empty path, and union arms other than the first, which the facts table doesn't hold yet. The audit's full-layout check adds them.
• Whether the two old models that were never registered were meant to be live. They're converted either way.