Message audit design - NexusTomorrow/server GitHub Wiki

How we establish exactly how to talk to the 16042 client, for every message it registers, before any feature code depends on one. It pauses the codec's third stage on the Roadmap, and builds on the declarations and facts check described on Codec design. What the server does with each message is out of scope: that starts with the core skeleton.

We assume every message, opcode and field we inherited is wrong until the client says otherwise, and check every one of them before more feature work. Each declaration records how far it has been verified, a test holds it there, and nothing below "fields verified" may be used. Until then a message carries a neutral name, and the results are published here as a protocol reference generated from the declarations.

A misread is silent. When the server sends a message the client reads differently, the client takes the wrong bits or the wrong meaning and carries on, and the symptom shows up far from the cause. The only defence is to know each message before anything depends on it.

The old code is wrong at every level. Measured against the client, it has a mistyped opcode (0x776E for 0x076E), one that isn't a client opcode at all (0x0511), a swapped pair (0x0851 and 0x0852), three messages declared in the wrong direction, 116 disagreements on the empty path alone, two reordered field pairs, nine command fields typed as integers where the client uses floats, and a vector named Rotation that's a start position. The layout check catches only some of that, and none of the meaning.

A name is a claim. The old code called 0x0111 ServerItemAdd, and a name like that tells everyone after what the message is for, whether or not anyone checked. Every name we inherited was a guess of that kind, so a name has to be earned the same way a width is.

Each message climbs five rungs, and reaches one only with every rung below it.

Rung	Name	Established when we know
R0	Identity	The opcode exists, and which way it travels
R1	Layout	Every field, array element, union arm and string, in order, at its width
R2	Purpose	What the message is for, and so its name
R3	Fields	Each field's meaning: its unit, range or set of values, the table an id indexes, integer or float, signed or not
R4	Conversation	When it may be sent, what has to come before it, and what the client does or sends in reply

R0 and R1 are checked mechanically, for every message at once. R2 and R3 are established for every message before more feature work, and each one also records the message's own side of the conversation: what state the client checks, and what it sends back. R4 needs the whole flow, so it's established flow by flow, milestone one's first; a message reaches it when every flow it takes part in is done.

Only what's established reaches a declaration. A claim that isn't settled holds its message below the rung it would complete.

A verified declaration carries the client's own names for the message, its nested types and its fields, and its rung in a [Verified] attribute beside [WireMessage]:

Where	Says
The type's <summary>	What the message is for
Each member's <summary>	The field's meaning, unit and range
<remarks>	The conversation, where it's known
[Verified(Rung.Fields)]	The highest rung reached: Purpose, Fields or Conversation. No attribute means R1 at most, since R0 and R1 come from the facts check

An enum-typed field gets its enum back only once its set of values is confirmed, and the enum's member names are checked against the values the client tells apart.

RegistryVerificationTests, beside the facts check and in the same shape, built at fadf745. Each run reports how far the audit has got; at fadf745 it reads "927 declared messages: 0 read to fields; 927 unverified."

• Every declared message is at Fields or above, or listed in a pinned Unverified set. A listed message that reaches Fields fails as stale, so the list only ever shrinks. It starts as every declaration and ends empty.
• A message at Purpose or above has a name of its own and a <summary>. Its neutral Client, Server or Message number gives way to a name for what it does. At Fields or above, every wire member of every layout it reaches has a summary, whatever the member's accessibility and whichever model project holds the layout, and a member still named Unknown<n> begins its summary with "The client gives this value no meaning." One sentence covers both directions: a value the client ignores when it reads, and one it writes with no meaning.
• Once the core skeleton adds dispatch, nothing below Fields can be used: registering a handler for one, or sending one, fails a test over the dispatch table. Until then the shrinking list is the gate.

The build writes XML documentation for Network, Network.Auth and Network.World, and the test reads it. CS1591 is off for the wire declarations, because the test enforces documentation by rung instead, and a message nobody has checked shouldn't be made to describe itself.

Everything not yet verified gets a name built from its position, so nothing in the code claims a meaning nobody has checked, and anyone can check a name without a lookup table.

Thing	Neutral name	Example
Opcode, one way	Client or Server and four hex digits	Client0094, Server01C2
Opcode, both ways	Message and four hex digits	Message0635
Message type	Its opcode's name	Server01C2
Nested struct	Its first holder (the lowest opcode), Part, and its order of first appearance	Server01C2Part0
Union interface	Its holder, Union, and its order	Client01C3Union0
Union arm	Its union, Arm, and the selector value in hex	Client01C3Union0Arm05
Field	Unknown and its position in the declaration, from 0	Unknown0, Unknown1
Enum-typed field	The smallest unsigned integer that holds its width	[UInt(5)] public byte Unknown3
Pad	A field like any other, since a pad claims the client ignores those bits	[UInt(32)] public uint Unknown4

The old code's comments go too. A comment on a model or a member is a claim of meaning just as a name is, so a declaration stays bare until it's verified.

What stays:

• Encodings: [Float32], [PackedFloat], [SInt(n)], the string kinds, arrays, unions and blobs. They change how our code reads the bits, not which bits travel, and each one is confirmed or corrected at R3.
• The codec's value types: NetworkGuid, Vector2, Vector3 and Quaternion, which are shapes in the codec, not models.
• The feature folders under Message/Model/, which group and claim nothing. A message moves folder when it turns out to belong elsewhere.
• Game.Static's enums, which serve the game side and the tables. The wire stops using them until a field's values are confirmed.

The opcode list is exactly the client's: 1,119 wire opcodes, the 181 the old list lacked among them under neutral names. Members the client doesn't register, State and State2, are gone with their declarations, and the 13 registrations at 0x10000 and up aren't wire ids and stay out.

Today's facts table holds each message's widths on the empty path: every count at zero, every union at its first arm. R1 needs the whole layout.

• The facts table grows into a tree: fields, arrays with their element layout, and unions with an arm for each selector value. It's still our own numbers, and still holds for both the 64-bit and the 32-bit client.
• The generator's layout grows into the same tree, emitted for every declared message. Built at 9f13a55, in the notation on Codec design, and held to the empty path the facts check already proves.
• The facts check compares trees, field by field, element by element and arm by arm. A declared arm the client lacks, or a client arm the declaration lacks, is a disagreement, and every disagreement is pinned by opcode, side and path with its reason, as today.
• Every registration gets declared. Each one with no declaration gets a neutral one generated from its tree, so the audit starts with every message declared and checked at R1.

What R1 can't see: fields of equal width swapped, a float against an integer, signed against unsigned, and meaning. Those are R3.

Messages are checked one feature area at a time (chat, guilds, housing and so on), because a feature's messages share code, tables and behaviour, and what's learned about one informs the rest. Milestone one's go first: pregame and realm, character select, world entry, entity commands and movement, and chat. Where one area ends and the next begins is still being settled, since the first area's messages don't all fall in one block of opcodes. An area's name groups messages; it doesn't claim anything about what they do.

Each area is reviewed on its own before its page of the protocol reference is generated again.

A tool in the repository, Tools/NexusTomorrow.ProtocolReference, loads the built Network, Network.Auth and Network.World assemblies, walks each one's generated WireMessages registry, reads the XML documentation the build writes, and generates this wiki's message pages:

• Messages: an index of every message by opcode.
• Messages-<area>: one reference page per feature area, with a section per message giving its opcode, direction, rung and purpose, a table of its fields with width, encoding and meaning, union arms and array elements nested under their holders, and the conversation in prose where it's known.

A message below Purpose shows its opcode, direction and layout only, marked unverified. The pages are generated, so they follow Documentation standards exactly and are never edited by hand.

1. Delete the old code. The proof against the old models paired declarations with them by name, so a rename would break it, and agreement with the old code is exactly what the audit stops trusting. Done at c5b0774.
2. The neutral rename: the names, then each file renamed for its type, then the enums the wire no longer uses. Every layout is bit for bit what it was. Done at 86b7f28, and the [Verified] attribute and its test followed at fadf745.
3. Bring Network.World and Network.Auth under the analysers. The rename had already cleared every naming finding, and the rest were 388 members with no initializer and four public fields, so every declaration the audit touches from here lands under the analysers and nullable reference types. Every layout is bit for bit what it was. Done at 9c93c58.
4. R1 for everything: the full-layout facts, the layout tree in the generator, and the tree check. The facts check's remaining pins fold into it. Under way: the tree notation and the generator's trees are in at 9f13a55.
5. R2 and R3 for every message, area by area, milestone one's areas first, and R4 flow by flow.
6. Close: a review of the whole audit, the protocol reference published, and the codec's third stage closed. Then build step 3 starts.

1. Every inherited opcode and model is assumed wrong until the client says otherwise.
2. Every message is checked up front, for its purpose and its fields, before more feature work.
3. The verification level lives in each declaration, and a test enforces it.
4. Everything unverified gets a neutral name: messages, nested structs and fields. Enum-typed fields become plain integers until their values are confirmed.
5. The codec's third stage pauses for the audit.
6. The results become a developer reference on this wiki.

• Nothing in the design. The work is the open part, and the Unverified count shows how much is left at any point.

Idea	What killed it
Checking only the messages the core uses, when it first uses them, with the mechanical layout check for the rest	Every message is checked up front instead, so no feature is built on a message nobody has checked, and the whole protocol is known before step 3
Recording each message's verification level in a separate list, held by review	The level lives in each declaration with a test, so the build holds it, and anyone reading the code can see what's trusted
Keeping the old code's names as labels until a message is checked	Neutral names instead, so nothing public asserts a meaning nobody has checked, and no check starts out anchored on the old code's guess
Renaming only the messages, or only the messages and their fields	Everything goes neutral, nested structs and enum-typed fields included, for the same reason
Finishing the codec's third stage first, then auditing	Once the proof against the old code was deleted, the rest of the third stage would have rested on the empty-path check alone

Claim	Basis
1,132 registrations, the same in the 64-bit and the 32-bit client: 366 client to server, 718 server to client, 35 both ways, 13 that aren't wire ids	Established from the client
The empty-path layout of 873 of 927 declared messages agrees with the client; 55 disagreements are pinned	CI run in the header badge
Full layouts: array elements, every union arm, string contents	Not checked yet. R1
Names, field meanings, integer or float, value sets	Not checked yet. The old code's guesses are out of the code since the rename at 86b7f28. R2 and R3
Every message is held to its rung	RegistryVerificationTests in the CI run in the header badge: 927 declared, none read to fields, all 927 pinned as unverified
Order and preconditions for milestone one's world entry and its other three flows	Established from the client; not yet seen in game, which happens once a server speaks them
The old code's errors listed under Why	Each found against the client; the opcode errors and swapped pairs are already corrected

Each new test is shown to fail the way it should before it's trusted: a run with one rung lowered, one summary removed, and one arm dropped from a facts tree.

• Whether the full-layout facts reach every path, such as a union selector wider than 8 bits or an array gated by a flag. A path found later is added to them, and to the tree check.
• How a feature area's boundary is drawn, and the order of the areas after milestone one's.

Piece	Path
The opcode list	Source/NexusTomorrow.Network/Message/GameMessageOpcode.cs
The declarations	Source/NexusTomorrow.Network/Message/Model/, Source/NexusTomorrow.Network.Auth/, Source/NexusTomorrow.Network.World/
The client facts table	Source/NexusTomorrow.Network.Codec/Facts/wire-facts-16042.jsonl
The facts check	Tests/NexusTomorrow.Network.Codec.Tests/Facts/
[Verified] and Rung	Source/NexusTomorrow.Network/Message/VerifiedAttribute.cs
The verification check, its pinned list and its rules	Tests/NexusTomorrow.Network.Codec.Tests/Verification/
The generator's layouts	Source/NexusTomorrow.Network.Codec.Generator/