Video Stream - HU-ICT-LAB/WebVR-Demo GitHub Wiki

Introduction

To be able to see what you are doing with the robot arm we would like a video stream from the webcam to A-Frame. This might sound simple, but we encounter a few problems.

  1. html5 only supports a limited amount of streaming protocols
  2. It has to be secure as the game is hosted using https
  3. We need to apply the video on an entity in A-Frame
  4. We need to start the actual stream.
  5. We need little to no delay

Solution

We are currently making use of a technology called WebRCT. WebRCT is a protocol commonly used for video or audio calling. A well know program that uses this software is Discord. We decided to use WebRCT as its a streaming format with a very low delay, as it is focused on video calls wich need to happen in (almost) real time. This takes care of our delay problem.

How do we stream WebRCT?

There are a few pieces of software we need to be able to stream WebRCT. The main piece is the OvenMediaEngine(OME). OME is a media streaming server that can receive and distribute video streams in multiple formats, it can be compared to a streaming service like twitch, you stream to the server and the server distributes it to the viewers.

A key feature of OME is the possibility to distribute the video using the WebRCT format. We can stream to the media server using https like we would stream to twitch, and OME will provide an WebRCT stream to our video feed. Another great feature is that OME is available as a docker container. This way it does not need a complicated installation or any dependencies except for docker, its basicaly plug and play. This takes care of problem 4.

How do we use OME?

At first we wanted to run OME locally on a pc. This is sadly not possible because of security restrictions. We need to view the WebRCT stream over a secure connection as the website is hosted on an HTTPS source. To do that we need certificates that are accepted by OME and correspond with the actual domain name the server is hosted on. Otherwise the server will fail to initialize the Secure streams.

To solve this problem we decided to host the server on the VPS (virtual private server) of floris. The installation is quite easy. You install docker on the system in question, and run the following command in the terminal:

docker run -d -p 1935:1935 -p 4000-4005:4000-4005/udp -p 3333-3334:3333-3334 -p 3478:3478 -p 8080:8080 -p 9000:9000 -p 9999:9999/udp -p 10006-10010:10006-10010/udp -v ~/server_files/:/opt/ovenmediaengine/bin/origin_conf -v /etc/letsencrypt/archive/florisvideler.nl:/etc/letsencrypt/live/florisvideler airensoft/ovenmediaengine:0.12.9

lets break this command down:

docker run this commando lets docker now that you want to create/launch a new container

-p 1935:1935 -p 4000-4005:4000-4005/udp -p 3333-3334:3333-3334 -p 3478:3478 -p 8080:8080 -p 9000:9000 -p 9999:9999/udp -p 10006-10010:10006-10010/udp these argument lets the docker container know which ports to open up for access to the streaming service. Make sure these ports are opened in the fire wall

-v ~/server_files/:/opt/ovenmediaengine/bin/origin_conf This argument mounts the content of the ~/server_files/ folder into opt/ovenmediaengine/bin/origin_conf on the docker container.

-v /etc/letsencrypt/archive/florisvideler.nl:/etc/letsencrypt/live/florisvideler This argument mounts the lets encrypt certificates of floris into the container at /etc/letsencrypt/live/florisvideler so these can be used to create a secure connection.

airensoft/ovenmediaengine:0.12.9 the last argument lets docker know wich image you would like to use. If this image is not found localy it will download it from the docker servers.

To configure the server to use TLS we need to provide a server.xml, this is a configuration file that can be used to tweak the server just the way you like it. This is also the file that is present in the ~/server_files/ folder. Using TLS for the server takes care of our security problem.

How do we view the stream?

To be able to view the WebRCT stream we make use of the OvenPlayer by OvenMediaEngine. This player is a pretty standard HTML5 video player, but with a few tricks we can make it work in vr.
To be able to watch streams in vr we need to have the video source available in a element in . To do this we generate the ovenmedia player and add it to a-assets. When we setup the player with the correct information it will eventually generate a component as a child entity inside . We can access this entity and give it an ID.

We now have the id of the video entity. We can then use this id to get the video feed from the component and attach it as a source to our entity. We can then view the video stream in vr! This takes care of problem 1 and 3.

All the code to make this possible is combined in our webrct_stream component and a-webrct-stream entity. This component makes sure that if you have multiple entities/components that want to watch the same stream only 1 videoplayer will be initialised for the source. You can use the component/entity like this:

<a-webrct-stream apply_fog="false" src='wss://florisvideler.nl:3334/app/stream' position="0 0 -9.9" width=16 height="9"></a-webrct-stream>

apply_fog: Set to "false" if you do not want the fog in the scene to effect the video.
src: The source video stream in WebRCT format (we can use other video formats but we will need to change the setup of the oven player.
position: The position of the entity
width: The width of the entity
height: The height of the entity

Other solution for LAN

Another option for video streaming is HLS developed by Apple. HLS is short for HTTP Live Streaming. Its a simple protocol that makes use of your already existing HTTP/HTTPS server to supply the stream. Its easily supported on LAN only systems and online systems. This format is not supported by default in most browsers though. For that you can use the hls.js library. This library allows you to fetch a HLS file (.m3u8) and apply the stream to a html5 tag/element.

You can then use this video element as a source for our objects or as a material for our entities.

This already takes care of the first 3 problems.

  • You can support HLS in firefox using hls.js
  • It is secure because its not a link/stream but a file in a folder on our own server, served using the same HTTPS python server
  • You can apply the stream to a html5 tag wich you can use to view the stream thru a entity or other entities using a texture.

How do we watch it in A-Frame

A-Frame and HTML5 dont support HLS out of the box. For this you have to use a JavaScript library that lets us do this. This library is called hls.js.

They have a cdn for it but this gives an error about hls.js.map not exisiting. So we downloaded the dist/hsl.js file from an example and removed //# sourceMappingURL=hls.js.map from the bottom of the file.

Next you can just include this file like an html script. Using the following code you can then display an HLS stream in a-frame on a enitity.

<html>
    <head>
        <script src="https://aframe.io/releases/1.2.0/aframe.min.js"></script>
        <script src="../scripts/hls.js"></script>
        <script>
            AFRAME.registerComponent('hsl_stream', {
                init: function(){
                    var video = document.getElementById('webcam_stream');
                    if(Hls.isSupported()) {
                    var hls = new Hls();
                    hls.loadSource('https://ip:port/path_to_hls_file/filename.m3u8');
                    hls.attachMedia(video);
                    hls.on(Hls.Events.MANIFEST_PARSED,function() {
                        video.play();
                    });
                }
                else if (video.canPlayType('application/vnd.apple.mpegurl')) {
                    video.src = 'https://ip:port/path_to_hls_file/filename.m3u8';
                    video.addEventListener('loadedmetadata',function() {
                        video.play();
                    });
                    }
                }
            })
          </script>
    </head>   
    <body>
        <a-scene cursor="rayOrigin:mouse">
            <a-assets>
                <video id="webcam_stream" crossorigin="anonymous"></video>
            </a-assets>

            <a-video src="#webcam_stream" width="16" height="9" position="0 4.5 -20" hsl_stream></a-video>

          
            <a-entity id="camera"  wasd-controls camera look-controls></a-entity>
            <a-sky color="#111"></a-sky>
        </a-scene>
    </body>
</html>

How can you stream HLS?

You can stream your camera using OBS. In OBS you can set a custom recording option, here you select FFMPEG and use HLS as our output. We lower the keyframe interval to 2*fps and done. We can then use the .m3u8 file generated by OBS to watch the stream.

The delay of HLS

A limitation from HLS is that the delay is pretty of high, we cant really lower it below 7/8 seconds because every HLS segment is 2 seconds and it has a segment list length of 4. This is usualy the default for HLS and it doesn't look like we can change this. There is a possibility to look into LLHLS or LL-DASH but those are quite a bit more complicated.

Something we stumbled upon during our research

We do not have cross origin policies set in our python server. As it is a simple http python server with an ssl wrapper for the socket. We learned that if we want to access files from another server we would need to add cross origin policies (CORS). We can do this by replacing the SimpleHTTP request handler with an extended class:

class CORSRequestHandler (http.server.SimpleHTTPRequestHandler):
    def end_headers (self):
        self.send_header('Access-Control-Allow-Origin', '*')
        SimpleHTTPRequestHandler.end_headers(self)

httpd = http.server.HTTPServer(('0.0.0.0', 8000), CORSRequestHandler)

But as this might increase security issues we decided to not add the policies. As the HLS files are on our own server, we dont need to allow get requests to other servers.

other solutions we tried:

RTSP to HSL

We have also tried starting a rtsp stream using https://github.com/aler9/rtsp-simple-server#publish-to-the-server. This is the protocol that IP cameras use. And then convert that stream to HLS using ffmpeg, this sadly didnt work yet.

VLC HTTP stream

We tried setting up a http stream using VLC media player. But as it was not secure the browser didn't allow access to the stream.

Other protocols using VLC and OBS

We tried many of the protocols supported by VLC and OBS but none seemed to work. We don't know why yet, but it probably has to do with html5 not supporting these streaming protocols. We tried UDP, SRT, TCP, RTP and RTSP

sources:

⚠️ **GitHub.com Fallback** ⚠️