Technical info for BrowserServer (Server part of WMBS) - mediaexplorer74/Windows-Mobile-Browser-Streaming GitHub Wiki

Technical info

Project Details

BrowserServer (Server part of WMBS) is a server-side project for Browser-as-a-Service application (WMBS).

Although you can configure some settings via the settings.xml file. You may update some settings such as port numbers and the CefSharp cache path to let run multiple solutions on same (one) device.

BrowserServer is under development. The main goal is to support some login-specific apps (i.e, SoundCloud, WhatsAppWeb).

Help the project growβ€”report issues and suggest new features! Note that the ultimate aim is to create a shared server for fans of legacy mobile systems, bringing them back to life, even if only briefly. If you have ideas on how to achieve this, start a discussion outlining your proposal.

WMBS is an open-source solution made with C#. Developers, you may use it freely, provided you follow these simplest guidelines:

  • fork repo and experiment with src code
  • reference to "previous authors" of this wonderful thing :)

API Explanation

Data Types and Their Purpose This section lists the standard data types from the System.Net.WebSockets namespace in .NET Framework 4.8. For more details on available data types and their purposes, see the official documentation. For simplicity, the System.Net.WebSockets prefix is omitted in the following examples.

When connecting to the main server:

WebSocketMessageType.Binary is used to send video when a Paint event fires or upon a user request for a page screenshot.

Data is sent as an unmarked byte array.

Data may be compressed with GZip or sent uncompressed, depending on the VideoStreamSettings/UseVideoGzipCompress setting in settings.xml.

WebSocketMessageType.Text is used to send textual messages. Text messages contain JSON data, which may include browser state information and some marked-up images.

When connecting to the audio transmission server:

WebSocketMessageType.Binary is used to send audio data.

Data is sent as a marked byte array.

Regardless of settings, each packet always includes a fixed-size 12-byte header consisting of two fields:

uint (4 bytes) β€” packet sequence number (32-bit integer)

ulong (8 bytes) β€” timestamp in nanoseconds (64-bit integer)

After the header comes the stream of PCM audio data.

The sample rate may be reduced to save data under the AudioStreamSettings/AudioSampleRate and AudioStreamSettings/UseAudioResampling settings in settings.xml.

Data may be compressed with GZip or sent uncompressed, depending on the AudioStreamSettings/UseAudioGzipCompress setting in settings.xml.

Text Data Packet

A packet has the following format:

{
    PType: int32|string,  // packet type
    PData: string,        // packet data as a string: JSON with browser state or marked-up image, or a plain text message
}

The PType field may take the following values depending on the packet’s destination.

If sent from server to client and PType is int32:

0 (NavigatedUrl) β€” current URL info

1 (TextInputContent) β€” current content in the active input field

2 (TextInputSend) β€” sent input text (not used)

3 (TextInputCancel) β€” no data; indicates the field no longer exists or is inactive

4 (LoadingStateChanged) β€” page loading state (LOADING or COMPLETE)

5 (OpenPages) β€” open pages list; entries separated by ;, each entry: Title|Id

6 (EditOpenTabTitle) β€” current tab title; format Id|NewTitle

7 (IsClientCanSendGoBackRequest) β€” true/false indicating back navigation availability

8 (IsClientCanSendGoForwardRequest) β€” true/false indicating forward navigation availability

Custom PType values without numeric IDs (they are string):

FullPageScreenshot

ChunkIndex (int) β€” zero-based chunk index

TotalChunks (int) β€” total number of chunks

Data (string) β€” Base64 image data (no GZip)

TabScreenshot

TabId (long) β€” target tab ID

Data (string) β€” Base64 image data (no GZip)

Note: PType may be either int32 or string. Check the data type to avoid processing errors.

FullPageScreenshot and TabScreenshot have no compression settings, but you can adjust image quality via VideoStreamSettings/FullPageScreenshotQuality in settings.xml.

If sent from client to server [PType is always int32]:

0 (Navigation) β€” send a URL or search text; prefix with skipchk: to skip URL validation. The BrowserSettings/SearchSystem setting in settings.xml controls fallback search engine.

1 (SizeChange) β€” resize browser window to new Size (Width, Height), multiplied by scalingFactor from VideoStreamSettings/ScalingFactor.

2 (TouchDown) β€” send a touch-down event as PointerPacket.

3 (TouchUp) β€” send a touch-up event as PointerPacket.

4 (TouchMoved) β€” send a touch-move event; PointerPacket has px, py, and id.

5 (ACK) β€” send ACK packet; server ignores it.

6 (Frame) β€” server ignores it.

7 (TextInputSend) β€” send text to active element and press 0x0D.

8 (NavigateForward) β€” navigate forward if possible.

9 (NavigateBack) β€” navigate back if possible.

10 (SendKey) β€” send a keycode (int) to the active input.

11 (RequestFullPageScreenshot) β€” request a full-page screenshot.

12 (ModeChange) β€” send Static or Dynamic mode.

13 (SetActivePage) β€” activate tab by ID and request its snapshot.

14 (GetTabsOpen) β€” request list of open tabs.

15 (CloseTab) β€” close tab by ID; if none remain, open FirstRunUrl (BrowserSettings/FirstRunUrl).

16 (RequestTabScreenshot) β€” request a screenshot of the specified tab.

17 (OpenUrlInNewTab) β€” open URL in new tab without validation.

18 (NewScreenShotRequest) β€” force send screenshot to client.

19 (IsCanGoBack) β€” request back-navigation availability; ignored by server.

20 (IsCanGoForward) β€” request forward-navigation availability; ignored by server.

21 (SendKeyCommand) β€” send a specific key action (Enter/Backspace).

22 (SendChar) β€” send a character via KeyCharPacket (JSONData, Shift, Ctrl, Alt, Layout).

Overview of Current Capabilities

Version 1.0.52.0 Streaming audio at 16 000 Hz via a custom IAudioHandler. Audio is additionally compressed using gzip and transmitted over a dedicated WebSocketServer, typically on port 8082.

Each audio sample is sent with an uncompressed header: an Int32 sequence followed by an Int64 ptc timestamp (in microseconds).

It is recommended to implement a custom JitterBuffer to track packet lag relative to the last received packet.

Below is an example client implementation for receiving and processing audio:

private void OnMessageReceived(MessageWebSocket sender, MessageWebSocketMessageReceivedEventArgs args)
{
    var reader = args.GetDataReader();
    reader.ByteOrder = ByteOrder.LittleEndian;

    if (reader.UnconsumedBufferLength < 12) return;
    int seq = reader.ReadInt32();
    long pts = reader.ReadInt64();

    uint payloadLen = reader.UnconsumedBufferLength;
    byte[] buf = new byte[payloadLen];
    reader.ReadBytes(buf);

    var msIn = new MemoryStream(buf);
    var gzip = new GZipStream(msIn, CompressionMode.Decompress);
    var msOut = new MemoryStream();
    
    gzip.CopyTo(msOut);
    var pcm = msOut.ToArray();
}

You can also visit the Line Browser repository for more details.

  • Sending browser screenshots when ChromiumWebBrowser.Paint is called. Screenshots are compressed with gzip, sent byte-by-byte, and unmarked.
  • Sending full-page screenshots as Base64 strings (marked).
  • Sending current URI information.
  • Sending information about open pages.
  • Sending information about a specific page title change.
  • Sending information about the text in the current input field.
  • Sending a signal if text input was canceled by the page.
  • Sending a text signal about page load status LOADING/COMPLETE.
  • Sending information about back-navigation availability.
  • Sending information about forward-navigation availability.
  • Receiving content from the client to insert into the current active element.
  • Receiving a request to send a full-page screenshot (static mode).
  • Receiving information about update mode Static/Dynamic.
  • Receiving information about sending a KeyCode to the active input element.
  • Receiving information about sending a Char to the active input element (preferred option).
  • Receiving information about action key presses.
  • Receiving information about the current active page.
  • Navigation. Use skipchk: to skip built-in URL validation.
  • Navigate back.
  • Navigate forward.
  • Resize browser window.
  • Handling TouchDown event.
  • Handling TouchUp event.
  • Handling TouchMove event.
  • Handling request to send all active tabs.
  • Handling request to send screenshot of a specific tab.
  • Handling request to close a specific tab.
  • Handling request to open URL in a new tab (no validation at this stage).
  • Handling request to send screenshot bypassing the Paint event.