Based on the shown need to include multiple services, dynamically instanced, within the same executable process; the demonstrated simplicity of the NeL Network Services' messaging and Naming Service mechanism; and the usability issues of the module system that was introduced alongside Ring; this document proposes a new networking protocol and complementary utilities to simplify the development and management of Ryzom services.

The current goal of this new protocol is to...

• multiplex multiple services, or interfaces, over each single connection,
• simplify the discovery of network services accross shard boundaries,
• add a security handshake,
• be more friendly towards modern web programming languages,
• remain a low overhead protocol,
• be implementable on top of different transport protocols,
• and provide a compatiblity bridge utility service between old services.

In the current NeL network services, each server executable contains a single service with a set of message callbacks it will respond to. Each service connects with the Naming Service in order to discover other services. Messages are sent simply by naming the target service, and providing the message name. The system is inherently low-configuration, as theoretically only the NS needs to be provided for a service to join the cluster. Received messages are buffered and processed every 100ms in the game loop.

A module layer was built on top of this, using interfaces generated from XML files, which effectively pipes an entirely new mechanism on top of a single message callback, also bypassing the NS, and introducing it's own connection mechanisms. This system is not as straightforward, and is complicated by proxying through the older system, but shows the need for services to expose separable modules, or interfaces, rather than being single instanced. Additionally, allowing multiple services to be exposed over a single connection also allows transparent proxies to expose subsets of services to other remote connections.

Generally, each channel on the connection is assumed to be a single stream with TCP-like guarantees. Independent messages are sent sequentially over the same single channel. In some cases, in case the QUIC protocol would be used, individual messages could be given an out-of-order flag to specify that their message chain may be on a separate stream from the channel itself. When using WebSocket as the backing protocol, each message and channel is transmitted over a single WebSocket connection.

For reusing channel numbers, a good strategy is that when a channel number is generated that is already in use, to skip forward by a random value to skip possible blocks of used numbers. Need to make the channel number range configurable for testing purposes.

The service connection is assumed to be unencrypted. For validating connection rights between services. The connecting service sends a random symmetric key, encrypted using the public key of the auth server. The connecting service also sends its connection passphrase, encrypted using the previously symmetric key. The auth service responds with a signed connection token that contains the connection rights of the service, encrypted with the symmetric key. The auth service does not assume that the connection is authenticated, the connecting server needs to use the decrypted token with other services. The signed token can be verified using the auth service public key.

The auth utility service will be advertised by the naming utility service to unauthenticated services. A service will reconnect to the NUS with the authentication when it has been granted.

AUTHS, 'authserv', nel_auth_service.exe

Upon disconnection from a naming service the existing cluster is retained, and upon reconnection the difference in reported state is pushed through. That is, when the naming service is disconnected no UP or DOWN messages will be received for services. A single ALL message is sent upon connection to sync the entire name set.

A service may programatically connect to multiple additional naming services, to allow connecting to remote namespaces. One namespace will be the default, and does not need a name, the others must be accessed by name.

Naming service assigns a numeric service ID for each service, and one for each host. The numeric ID is used as a single integer composed of two 16bit integers. A service ID by itself is unique, but the host ID is always included, to reduce the chance of re-use conflicts.

NUS, nel_naming_utility_service.exe DS, 'discover', nel_discovery_service.exe

Effectively pretends to be the legacy NS for old services, proxies connections for new services. Possibly could also proxy the module stuff, but unlikely. Legacy services will see incoming messages as coming from BUS. Services may subscribe a message identifier prefix with BUS, that legacy services may use to send to new services through BUS. Or may be separate from legacy NS and actually set up a tcp listener for each service that wants to be advertised.

BUS, nel_bridge_utility_service.exe

Executable that combines all utility services in one, simplifying debugging.

nel_utility_services.exe

• IFrameStream: General interface for any two-way messaging stream. Usually subclasses will have a constructor with an IBlobStream or other object to which to forward data, and an IBlobStream which will receive data. receiveFrame, sendFrame
• Multiplexer: Implements the channel multiplexer. Multiplexes multiple IBlobStream over a single IBlobStream, two-ways. MuxConnection, MuxChannel
• Messaging: Implements the message chain and callback handling.

The base connection is any framed messaging protocol. The current target is WebSockets since it's ubiquitously available everywhere.

The message frame connection is multiplexed into channels for arbitrary usage. Both sides of the connection can arbitrarily open new channels. When opening a channel, an arbitrary payload blob is included for the application to interpret and process. The payload can be used for authentication, identification, and addressing purposes.

These multiplexed channels are effectively a framed messaging protocol, and technically can be recursively multiplexed into more channels.

An entire channel can be transparantly proxied to another host without parsing it's contents.

A message chain protocol is defined to run on top of any framed messaging protocol. This is similar to an RPC protocol, but rather than being purely request-response, each message can be a response to any previous message, and multiple messages can be sent as a response to form a streamed response. The message chain can recursively respond with streams to individual stream responses, and so on.

An entire message chain starting from any message can be transparantly proxied to another host without parsing it's contents.

Network protocol designed for structuring microservice server-to-server and server-to-client communications over WebSockets in an RPC-compatible fashion.

It's purpose is to establish a near-0-configuration network between microservices. (The only configuration parameter that needs to be set is the address of the discovery services and a security key.)

The design is loosely based on NeLNS, using WebSockets as the frame transport layer to take advantage of existing HTTPS routing mechanisms and existing compression implementations. Technically we may design a more lightweight frame layer directly on top of TCP and TLS to bypass the WebSocket design, but this does not seem necessary at this point.

The service network is composed of services where each service has a globally unique service identifier. The unique service identifiers are obtained from the discovery service.

Service identifiers in string format are as /service/a1b2, where in binary form the first part is a UTF-8 string of up to 8 bytes, and the second part is a 6 byte 48 bit integer LE. The integer part in text form is hexadecimal, and may not have leading zeroes.

In case a generic service is requested, the identifier omits the integer part as /service, and the integer part will be 0. Specifying /service/0 must be permitted, but should not be used.

The connection is multiplexed at the primary layer by means of channel identifiers. (Comparable to namespaces in sockets.io.) Available channel numbers are numbered from 0 to 0xFFFFFFFFFF (that is, a 48 bits value). The client may assign any even numbers, the server may assign any uneven numbers. (Similar to HTTPS/2 stream numbering scheme.) Channel numbers may be selected arbitrarily but (for now) may not be reused out of security precaution. When running out of channels, a new TCP connection should be established for further communications. Channel 0 and 1 are reserved, and may not be used.

The channel feature is intended to allow a service to connect to another service through a dispatching end point (which may be a load balancer or other routing utility service,) and to allow multiple virtual services to exist within one process (that is, it allows connecting through multiple services over a single TCP connection.)

At transport level, opening a channel is immediate, closing a channel is fully effective once both sides have confirmed closing the channel. Once a channel has been closed, you may no longer send to it, once a channel has been remotely closed, you may no longer receive messages from it. A locally closed channel that is still open remotely can still receive messages.

At the switchboard level, creating a channel implies requesting for availability of a specific service by either it's unique identifier or by a generic identifier. A channel that has been closed locally will still accept incoming responses, will but drop incoming messages, and will abort any incoming request messages.

Requests and their associated responses are abstracted by a highly flexible chaining mechanism.

Messages which do not expect any response are sent without request identifier.

Requests which expect a response are sent with a request identifier. The request identifier is a number from 0 to 0xFFFFFF (that is, a 24 bits value), which is local to the sender, which may be selected arbitrarily, and which may be re-used when the request has been serviced.

Responses contain the sender's request identifier as response identifier. A response may include a request identifier in turn, if it expects a further reply, effectively allowing chained (or trampolining) procedures to be implemented between services with minimal complexity on tracking state.

Messages which are not responses must include a procedure identifier. This is an 8-byte integer, composed from a short UTF-8 string of up to 8 bytes. This identifier specifies which procedure in the receiving service will handle the message. Responses may omit the procedure identifier. A procedure identifier may also be an arbitrary length string, but not all implementations are required to accept this format, due to platform limitations.

Messages with a known but unauthorized procedure may be queued until authorized (or until time-out.) In the API, authorization sets are applied to channels to specify which procedures are permitted for a given channel.

Connections and channels are handled by a central Switchboard object. The local switchboard is provided a single authorization token which is to be used accross the entire system. This authorization token is not an identification token, and merely gives the node a scope of access. The permitted access to the network is configured on the discovery service. Updating the authorization token updates it accross all channels.

A service may set up a stateful channel to another service, or may use channels managed by the switchboard to send one-off messages (10s timeout on the connection for re-use), or to locally handle load balancing or sharding.

To create a stateful persistent connection, the client issues switchboard->openChannelUrl("wss://hostname/#/service/a1b2"); (or without url as openChannel(host, service, instance)). This uses an existing connection to wss://hostname/ if available. The autorization information includes the clients's hostname on the network, so the created connection can be used to open channels both ways. A hostname may include a port number, and a path for routing through 3rd party HTTPS endpoints. The hostname and path are ignored by incoming connections, it is assumed they may be routed as-needed. Once a connection is available, the bridge will open a new channel to /service/a1b2 on the target host.

The switchboard may be connected to the discovery service, but this is not a hard requirement. Generally all the backend will have access to the discovery service, and end-user clients will connect to a single end-point host, and not be indexed by the discovery service. When a service is connected to the discovery service, the discovery service and the service will have a synchronized table of services mapping to hostnames (as well as any sharding slot tables.)

So, the switchboard may be configured in two modes. One, through the discovery service. Two, through the proxy or endpoint service.

When using the proxy service, a channel may be directly opened by name without global id.

There is only a single master discovery server, and there may be multiple slave discovery servers which act as passthrough and mirror .

Unique instances of services may be named, and accessed as wss://hostname/#/service/~name, this is only intended for development pu rposes.

If a client does not require a stateful connection, they may issue simplified commands as switchboard->sendRequestUrl("/service", 'GET_PROF', byte[]); (or without url as sendRequest(host, service, instance, procedure, data).) Parameters host and instance may be null.

Create the switchboard.

Switchboard^ switchboard = refnew TalkBridge();

Set the authorization token. This token specifies which services we may create and connect to. Discovery dashboard can generate tokens for a specified environment.

switchboard->setAuthorizationToken(...);

Connect to the discovery service. The hostname ds.example.com should point to a private address in production environments.

switchboard->connectDiscoveryServices({ "ws://ds.example.com/#/ds", ... });

Register the service.

ITalkService^ service = rc ITalkService(); // Create an instance of a service, which implements a channel factory.
switchboard->registerService('api', service);

MicroService may connect to other services, ITalkService implements those functionalities.

class MicroService implements ITalkService {
        /// Create a channel for between this service and the remote service that is specified by it's service identifier
        /// Remote service may be null if created directly from switchboard
        /// Remote instance may be null if created directly from switchboard, or if it's an anonymous proxied device
        /// TBD. authPayload
        override function ITalkChannel^ createChannel(uint service, uint remoteService, uint remoteInstance, byte[] authPayload) {
                return rc MicroChannel(this);
        }
}

class MicroChannel implements ITalkChannel {
        MicroService^ m_Owner;
        MicroChannel(MicroService^ owner) : m_Owner(owner) { }
        override function receiveMessage(uint procedure, byte[] data) {
                switch (procedure) {
                        // ...
                }
        }
}

ITalkService^ service = rc ITalkService(); // Create an instance of a service, which implements a channel factory.
switchboard->registerProxyService(service);

registerDefaultService

A proxy service is a service factory which acts as a 'catch all' for channels. This is useful for backend proxies.

To enhance authentication flow, rather than having a proxy channel be created entirely stateless, a proxy passthrough channel can be created as a child of a proxy endpoint channel.

For this, the provided proxy service base class has an overridable method inside the EndPointChannel, which allows creating generic proxy channels with specific user data attached to them, and any necessary filtering applied to messages.

In other words, for server-to-server proxying, the proxy channel is made directly on the switchboard. For server-to-client proxying, the proxy is arbitrated through the existing channel (and may actually be a channel to any local service, rather than a proxy.)

TBD. Need to split definition of endpoint (server-client) and tunnel (server-server).

A consumer client connects to an endpoint.

Create the switchboard.

Switchboard^ switchboard = refnew Switchboard();

Set the authorization token. This is a generic authorization token for all clients that allows access to just the endpoint. It does not and may not and must not identify the user.

switchboard->setAuthorizationToken(...);

Connect to the discovery service. The hostname api.example.com should be a redirecting load balancer (not a passthrough) which redirects any connection to an available endpoint (using HTTP redirect). The redirection mechanism may be behind a load balancer. The hostname api.example.com should be a load balancer to all the end points. Connection to an end point through the load balancer hostname will cause redirection to the actual direct public url of the end point, if configured to do so, to bypass the load balancer.

~~switchboard->connectEndPoints({ "wss://api.example.com/" });~~

Create a channel directly to an endpoint service. From the POV of the endpoint service, the remote client is an undefined service.

~~EndPointClient^ ident = await switchboard->createChannel('ident', channelFactory);~~

Or more simply without setting the end points.

EndPointClient^ ep = await switchboard->createChannelUrl("wss://api.example.com/#/ep", channelFactory);

Process any custom identification.

ep->login(...); // etc

On login, create the real channels, inheriting any state from the existing channel. The state inheritance is processed by the remote service, and is implementation-specific.

~~ident->createChannel('api', channelFactory);~~

Alternatively.

switchboard->setChannelBroker(ep);
ApiClient^ api = await switchboard->createChannel('api', channelFactory);

TBD. For client-side model, provide automatic reconnection mechanism. Re-establish login, etc.

OTOH. We may update the authorization token, and flag it as verified on the endpoint, then pass it as connection payload. Authorization is bound to a connection, not to a channel, but we may route it into a channel...

What's neato here, is that the autorization is bound to a connection, meaning if a channel is created it doesn't require additional authentication. We may, however, provide a means to push an alternative authorization onto a channel. The remote host will have to re-verify it then, though.

Proxy are handled on 'WSChannel' level, rather than on TalkChannel level.

While tempting to create magical generic interfaces for a service, the requirement is more often for specific interfaces for a service.

Channels are often not "between between a generic client and the service", but "between service A and service B".

Thus, a potential channel will most likely require specialized implementation to match A and B. However, a specialized local implementation may be transparantly generic on the remote side.

TBD. Uses a hash slot mechanism to easily scale services.

TBD. Designed to just store data, cached, sharded, with timeout or memory limit as needed. Prefers dropping data in favour of newer data when race conditions occur during reconfiguring.

TBD. Provide a neato dashboard node. Provides info on the current state of the discovery service, essentially. Could be used to fetch stats from all the nodes as well.

/// Root class. This lets you set up a listening connection, and connect to other nodes.
/// A service process creates one of these. Services instances register with the CSwitchboard.
class CSwitchboard
{
  /// Registers a service with CSwitchboard (for instance, LS).
	void registerService(IService *service);
  
  /// Connection managed here directly.
  void startListening(int port);
  void connect(const std::string &host);
  /// Or alternatively provide functions to push frames into the switchboard?
  void clientConnected(IFrameChannel *);
  void packetReceived(IFrameChannel *, IStream &packet);
  // etcetera..
  
};

/// Parent class of a service. This has some callbacks for new connections, etcetera.
class IService
{
  /// A new peer makes a connection with this service. Each connection is represented by a channel.
  /// Following the factory pattern, a new channel pointer is returned.
  /// This allows connection-specific state to be stored more easily.
	CSmartPtr<IChannel> channelConnected(IStream &payload);
};

/// Represents a single virtual connection to a service.
class IChannel
{
	function sendMessage(string name, IStream &msg);
  function sendMessage(int id, IStream &msg);
  function sendMessage(IStream &msg);
  function addMessageHeader(IStream msg, string name);
  function addMessageHeader(IStream msg, int id);
  // etcetera
  
  function receivedMessage(string name, IStream &msg);
  function receivedMessage(int id, IStream &msg);
  // etcetera
  
  function disconnect();
  function disconnected();
  // etcetera
};

Should ideally write it, so that it can be battle-tested without an actual connection...

We can easily pipe the legacy nel network service API on top of this through a generic NLNET service.

Similarly, this could be piped on top of the legacy nel network API...