Connection Protocol - TUMFARSynchrony/SynthARium GitHub Wiki

Contents

Introduction

This page documents the protocol used for client-server connections. This includes sending and receiving messages, sending a stream to the server and receiving streams from the server.

In case you are unfamiliar with WebRTC, a good entry point to get a basic understanding can be:

  • WebRTC.org: Getting started with WebRTC - a good overview over a typical WebRTC application flow
  • WebRTC on mdn: WebRTC documentation on mdn

We're using (half-)trickle ICE to speed up the connection process. You can read up on it here:

Approach

To allow the experiment hub to modify the streams (audio and/or video tracks) of its users, all streams must be send to the Server.

For each incoming stream, the server applies relevant filters, records and then distributes the stream to other users. This is mainly the task of the Connection class, with help of SubConnections to distribute incoming streams to other users.

Establishing an Initial Connection

Description

In this step, the initial connection is established.

The initial connection provides the following features:

  • Sending local tracks to the backend
  • Receiving remote tracks based on the ones send
  • Communications using a datachannel

Protocol

  1. [Client] Initialize new RTCPeerConnection

    • Create new RTCPeerConnection
    • Add a data channel
    • Add local (audio and video) tracks
    • Add required event handlers to RTCPeerConnection here
    • Generate uuid for connection

  2. [Client] Create offer and send request to Server

    • Create new WebRTC offer
    • Set offer as local description
    • Pack offer and additional parameters into request object:
      • sdp: Session Description Protocol (SDP) of offer created in previous step
      • type: type of the offer created in previous step
      • user_type: "participant" or "experimenter"
      • connection_id: uuid generated in step 1
      • If user_type is participant
        • participant_id
        • session_id
      • Else user_type is experimenter:
        • TODO: Experimenter authentication
    • Send request object to <server address>/offer
      • Body: stringified request object
      • Method: POST
      • Mode: If Environment is DEVELOPMENT, CORS.

  3. [Client] Send ICE candidates

    • Everytime a new ice candidate is gathered:
      • Pack candidate and id into request object:
        • id: uuid created in step 1
        • candidate
      • Send request object to <server address>/addIceCandidate
        • Body: stringified request object
        • Method: POST
        • Mode: If Environment is DEVELOPMENT, CORS.

  4. [Server] Handle offer and respond with answer

    • Ensure that contents of offer comply with the above definitions
    • Additional checks, e.g. if session_id exists or if participant_id is part of session_id.
    • Create new RTCPeerConnection
    • Add event handlers to RTCPeerConnection here
    • Set remote description of RTCPeerConnection to received offer
    • Create WebRTC answer
    • Set answer as local description
    • Send response to client:
      • On Error: Message containing Error
      • On Success: Message with type SESSION_DESCRIPTION:
        • sdp: Session Description Protocol (SDP) of the answer
        • type: type of the answer
        • Optional: participant_summary: Participant Summary object

  5. [Server] Handle ice candidates

    • Ensure that ice candidates comply with the above definitions
    • Associate candidate with corresponding connection using uuid
    • Add ice candidate to RTCPeerConnection
    • Send response to client:

  6. [Client] Handle answer

    • On Error: abort connection establishment -> connection failed
    • On Success: Set remote description to answer in response

  7. [Server] Handle incoming datachannel & tracks

    • This step refers to event handlers that should be set in step 4, after the RTCPeerConnection is initialized in the Server
    • Handle incoming datachannel
    • Handle incoming tracks and send them back to the client

  8. [Client] Handle incoming tracks

    • This step refers to event handlers that should be set in step 1, after the RTCPeerConnection is initialized in the Client
    • Tracks send to the backend should be send back now. Handlers for the track event set in step 1 will be called now

Note: Steps 2,4 and 3,5 happen at the same time and the order is not fixed.

Initial Connection Sequence Diagram Example

Initial Connection Sequence Diagram Example

Adding a Sub Connection

Description

When the client needs to receive an additional stream, a sub connection is started.

A sub connection provides the following features:

  • Receiving tracks from the backend

Sub connections should not include a datachannel, the initial connection is used for communication.

Protocol

  1. [Server] Create new SubConnection and notify client

    • Create RTCPeerConnection
      • Add required event handlers to RTCPeerConnection
      • Add video & audio tracks
    • Create ConnectionProposal Message and send it to the client
      • Endpoint / message type: CONNECTION_PROPOSAL

  2. [Client] Initialize WebRTC connection

    • Create new RTCPeerConnection
    • Add required event handlers to RTCPeerConnection here
    • Create ConnectionOffer Message and send it to the server
      • Endpoint / message type: CONNECTION_OFFER

  3. [Client] Send ICE Candidates

    • Everytime a new ice candidate is gathered:

  4. [Server] Respond with CONNECTION_ANSWER

    • Handle offer in RTCPeerConnection created in step 1.
    • Create answer
    • Set local description to answer
    • Create ConnectionAnswer Message and send it to the client
      • Endpoint / message type: CONNECTION_ANSWER

  5. [Server] Handle ICE candidates

    • Ensure that ice candidates comply with the above definitions
    • Associate candidate with corresponding sub connection using subconnection id
    • Add ice candidate to RTCPeerConnection
    • Send response to client:

  6. [Client] Handle answer

    • Set remote description to answer

Notes

  • Step 1 / the CONNECTION_PROPOSAL ensures that the client always initiates the connection. This can help with ICE.
  • Steps 2,4 and 3,5 happen at the same time and the order is not fixed.

SubConnection Sequence Diagram Example

SubConnection Sequence Diagram Example