Server architecture - rebaserHEAD/NexusForever GitHub Wiki

How the server is put together: nine processes, three client connections, a message broker between them, and one thread that runs the game. It's the map to read before changing anything. The process, port and project tables are on Server architecture processes and projects.

flowchart TD
    CL(["Client"]) -->|1. log in| STS["STS server"]
    CL -->|2. game token| AUTH["Auth server"]
    CL -->|3. session key| WS["World server"]
    WS <--> BUS{{"Message broker"}}
    BUS <--> SVC["Chat, group, friendship<br/>and character servers"]
    SVC --> API["Character and<br/>account APIs"]

The client only ever opens three connections, in order. The STS server checks the password and issues a game token, the auth server trades that token for a session key, and the world server accepts the session key and runs the game.

Chat, groups, friends and /who live in four smaller servers with no client connection at all. Whatever the client sends for them reaches the world server first and crosses the broker. Those servers look characters and accounts up through two small HTTP APIs.

Seven databases, each owned by one part of the server:

The rule underneath, from upstream's database style: account-wide data goes in auth, anything per realm goes in character, and static data goes in world.

Everything that touches game state happens on one dedicated thread. Each pass, in order:

1. Sessions. Every queued packet from every client is handled, then each session's outgoing packets are sent.
2. Maps. Every loaded map updates, each top-level map in parallel with the others, and the pass waits for all of them.
3. Everything else. Buyback, quests, guilds, housing, the login queue, matchmaking, scripts, shutdown, then GM commands.

Then it sleeps a millisecond and starts again. There's no fixed tick rate: a pass takes as long as it takes.

That shape buys a lot of simplicity:

• Handlers never race each other, or a map update, so game code needs no locks.
• A handler that blocks stalls the whole world, which is why a player save runs as a task instead of on the thread.
• A reply leaves on the next pass. A packet queued during the map step goes out at the start of the next one.

1. Socket reads happen on the thread pool. They only cut the stream into packets, using a 4-byte length prefix, and queue them.
2. On the world thread, each queued packet's 16-bit opcode picks its message model, and the model reads the fields.
3. The handler for that model is found by reflection: any class implementing IMessageHandler<TSession, TMessage> is registered at start-up, with nothing to list by hand.
4. Nearly everything the client sends arrives wrapped in an encrypted or packed envelope. The envelope's handler unwraps it and hands the inner packet back to step 2.

A packet with no model, and one with a model but no handler, each log their own line, which is how you find what isn't built yet.

A player is saved every 60 seconds, on first entering a map, on logout, on the character save command and at shutdown. A save pauses that player's incoming packets until the database write returns, so nothing they do can land halfway through it.

Nothing sends a message to a particular server. Each server publishes, and each one subscribes at start-up to the message types it wants. A "reply" is just a second message that the asking server happens to subscribe to.

• A player is addressed inside the message, by character id and realm. Every world server subscribed to that type receives the message, and the one that has the player online acts on it.
• The world server is the source of truth for who's online. It publishes logins, logouts, and changes to stats, zone, world and position, and each side server keeps its own copy.
• The side servers write through an outbox. Chat, group and friendship save outgoing messages in the same database transaction as the change that caused them, and a scheduled job publishes them. Chat and friendship run that job every second.
• The transport is a setting: Network:Internal:Broker picks RabbitMQ or Azure Service Bus.

Broker handlers run on the broker's own workers, not on the world thread. Code there that changes an entity should hand the change to the world thread with GridEntity.SynchroniseAsync. This is read from the code, not measured.

Quest, creature, map and instance behaviour can live in content scripts, compiled into separate NexusForever.Script.* assemblies.

• Script:Enable is the switch. It defaults to off in code and on in the example config. With it off, nothing that only a script creates, such as a public event, ever exists.
• A script binds by the owner it names. A class implementing IOwnedScript<T> attaches to every owner of type T, narrowed with [ScriptFilterOwnerId] or [ScriptFilterCreatureId]. With no filter, it attaches to all of them.
• Scripts only see interfaces. Content projects reference Game.Abstract, never Game, so anything a script needs must be on an interface.
• Script:Dynamic:Enable adds compiling from source, for script folders with no built assembly.
• Scripts reload while the server runs. When a script assembly changes on disk, it's unloaded and loaded again on the next pass.

Every manager is registered in the dependency injection container, and most code still reaches managers through a static Instance that resolves from that container. Upstream is partway through moving from the static route to constructor injection, so new classes take their dependencies through the constructor.