Home » MP4Client » Multiview rendering

GPAC supports streaming HEVC tiled DASH videos. In this page, you will find some helpful information to get started with this feature. In the following, we assume the input video has resolution of 3840×2160 and a frame rate of 30 frames/sec.

The open-source Kvazaar encoder allows encoding HEVC with motion constrained tiling. To instruct the encoder to encapsulate each tile in a separate slice, the --slices option is used with the value tiles. Motion vectors are then constrained within in each tile using --mv-constraint.

$ kvazaar -i input.yuv --input-res 3840x2160 -o output.hvc --tiles 3x3 --slices tiles --mv-constraint frametilemargin -q 30 --period 30 --input-fps 30

We can also set the target bitrate by --bitrate in bps instead of setting QP. It will turn the rate controller on. If you plan on merging tiles with different quality, keep the same QP settings and adjust the bitrate (changing QP would generate incompatible bitstreams). The output of the encoder is a raw bitstream and should be packaged within a container. To do so, MP4Box from GPAC can be used as follows, where the set value for -fps should match that of the encoded video.

$ MP4Box -add video_tiled.hvc:split_tiles -fps 30 -new video_tiled.mp4

The generated MP4 file includes one base track containing parameter sets and/or SEI messages plus one track for each tile. Now you can generate DASH segments and descriptor for the packaged video as follows.

$ MP4Box -dash 1000 -rap -frag-rap -profile live -out dash_tiled.mpd video_tiled.mp4

MP4Box can also generate an MPD with multiple representations by adding more input files.

$ MP4Box -dash 1000 -rap -frag-rap -profile live -out dash_tiled.mpd video_tiled_rep1.mp4 video_tiled_rep2.mp4

For more information, please refer to this article.

Once the content is generated and stored on the server side, you can stream it using MP4Client from the client side. The only required argument is the URL of the MPD file. In the case of 360/VR videos, it is possible to render only the viewport using #360LIVE right after the URL. For example:

$ m4pclient http://server:port/test.mpd
$ m4pclient http://server:port/test.mpd#360LIVE

The main loop to render the 360 video content (in #360LIVE mode) is roughly as follows.

gf_sc_draw_frame
   gf_sc_render_frame
      gf_sc_draw_scene
         visual_draw_frame
            visual_3d_draw_frame
               visual_3d_setup
               visual_3d_setup_traversing_state
               visual_3d_setup_clipper
               visual_3d_init_shaders
               for each view
                  visual_3d_draw_node // for root
                  for each scenegraph
                     gf_sc_traverse_subscene

gf_sc_draw_frame() function is called in MMLoop function. MMLoop is part of Media Manager module and is run by Terminal module in a separate thread.

Since HEVC supports tiling, putting the tiles back to form the whole frame is done inside the decoder. Therefore, the output of the decoder is the projected 2D video (e.g. equirectangular). The decoder puts each decoded frame as a composition unit (CU) inside the composition buffer (CB). The compositor takes out CUs one at a time (by calling gf_sc_texture_update_frame() function) and tries to push the texture to the graphics device (by calling gf_sc_texture_push_image() function).

The conversion from YUV (YU12) to RGB is done with the help of a fragment shader. visual_3d_shader_with_flags() function is used to load the shaders.

The video output interface (DirectX, SDL, or Raw) is loaded from a module in gf_sc_create() function and signaled to setup in gf_sc_reconfig_task() function.

TAG_MPEG4_OrderedGroup
   |
   |--- TAG_MPEG4_Background2D
   |--- TAG_MPEG4_Viewpoint
   |--- TAG_MPEG4_NavigationInfo
   |--- TAG_MPEG4_Sound2D
   |       |
   |       |--- Source: TAG_MPEG4_AudioClip
   |
   |--- TAG_MPEG4_Transform2D
           |
           |--- TAG_MPEG4_TouchSensor
           |--- TAG_MPEG4_Transform2D
                   |
                   |--- TAG_MPEG4_Shape
                           |
                           |--- Appearance: TAG_MPEG4_Appearance
                           |                   |
                           |                   |--- Texture: TAG_MPEG4_MovieTexture
                           |
                           |--- Geometry: TAG_MPEG4_Sphere