Render API - wang-bin/mdk-sdk GitHub Wiki

MDK supports rendering video via OpenGL, D3D11/12, Vulkan and Metal. You can choose which api to use at runtime via Player.setRenderAPI(RenderAPI* api, void* vo_opaque = nullptr).

OpenGL is the default api if you don't call setRenderAPI(). To use another api, a concrete RenderAPI subclass object is required. For example to use d3d11

    D3D11RenderAPI ra;
    // set ra.context, ra.rtv if provided by app
    player.setRenderAPI(&ra);

4 Rendering Modes in MDK

Render To Texture

Render target is set by user, we start render pass internally

OpenGL

No need to call player.setRenderAPI() if fbo is correct

    GLRenderAPI ra{};
    ra.fbo =;
    player.setRenderAPI(&ra, this);

D3D11

    D3D11RenderAPI ra{};
    ra.context = immediateContext; // immediateContext provied by app, can be null if rtv is not null
    ra.rtv = renderTargetView; // ID3D11Texture2D* or ID3D11RenderTargetView* provided by app, can not be null
    player.setRenderAPI(&ra);

To use the texture as video encoder/processer input, must ensure current draw is completed

IDXSync

class __declspec(uuid("1c5a6669-0875-4212-9c0f-afefb0482146"))
IDXSync : public IUnknown
{
public:
// sync to producer, wait on consumer.
    virtual HRESULT producerReady(UINT subresource) = 0;
    virtual HRESULT consumerWait(IUnknown* pCtxOrCmdQ, UINT subresource, DWORD cpuWaitMs = 0) = 0;

// sync to consumer, wait on producer.
    virtual HRESULT consumerRelease(IUnknown* pCtxOrCmdQ, UINT subresource) = 0;
    virtual HRESULT producerWait(UINT subresource, DWORD cpuWaitMs = 0) = 0;
};

    ComPtr<IDXSync> s;
    UINT size = sizeof(IDXSync*);
    // res is ID3D11Resource
    if (SUCCEEDED(res->GetPrivateData(__uuidof(IDXSync), &size, s.ReleaseAndGetAddressOf()))) {
        s->consumerWait(immediateContext, subresource/* usually 0*/);
    }

D3D12

    D3D12RenderAPI ra{};
    ra.cmdQueue = ;
    ra.rt = ;
    player.setRenderAPI(&ra, this);

Vulkan

    VulkanRenderAPI ra{};
    ra.device = ;
    ra.phy_device = ;
    ra.opaque = this;
    ra.rt = ;
    ra.renderTargetInfo = [](void* opaque, int* w, int* h, VkFormat* fmt, VkImageLayout* layout) {
        return 1;
    };
    ra.currentCommandBuffer = [](void* opaque) -> VkCommandBuffer{};
    player.setRenderAPI(&ra, this);

Metal

    MetalRenderAPI ra{};
    ra.texture = ;
    ra.device = ;
    ra.cmdQueue = ;
    player.setRenderAPI(&ra, this);

Examples:

Render in a Foreign Render Pass

In this mode, render pass and render target are setup in user code. It's named "Foreign Context". Usually you have to render a frame in a correct context in the rendering thread of the gui toolkit or app. For example, call Player.renderVideo() in QOpenGLWidget::paintGL() or similar events for Qt Apps, in GLSurfaceView.RendereronDrawFrame(GL10 gl) for android apps. The callback of Player.setRenderCallback(callback) will be called if a new frame is ready to display, you can notify your gui toolkit or app in this callback.

You need to call only

  • setVideoSurfaceSize(): frame buffer size. usually the window/surface size. call it when window/swapchain is resized
  • setRenderCallback(): the callback is called when a new frame can be rendered. you can notify your rendering thread to render a frame. NOT REQUIRED if rendering not driven by decoded video frames, e.g. Apple DisplayLink, Android GLSurfaceView, Qt VSync.
  • renderVideo(): record draw commands

OpenGL

No need to call player.setRenderAPI(), we can obtain the implicit global command queue(context)

D3D11

To use a RenderAPI other than OpenGL, a concrete RenderAPI subclass object with some valid members is required.

    D3D11RenderAPI ra{};
    ra.context = immediateContext; // immediateContext provied by app
    player.setRenderAPI(&ra);

D3D12

    D3D12RenderAPI ra{};
    ra.cmdQueue = ...;
    ra.colorFormat = ...;
    ra.opaque = this;
    ra.currentCommandList = [](const void* opaque){ ... };
    player.setRenderAPI(&ra, this);

Vulkan

    VulkanRenderAPI ra{};
    ra.device = ;
    ra.phy_device = ;
    ra.graphics_family = ;
    ra.graphics_queue = ; // optional but recommended
    ra.opaque = this;
    ra.render_pass = vkrpnat->renderPass;
    ra.renderTargetInfo = [](void* opaque, int* w, int* h, VkFormat* fmt, VkImageLayout* layout) {
        *w = ;
        *h = ;
        *fmt = VK_FORMAT_R8G8B8A8_UNORM; // MUST match render pass
        *layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
        return 2;
    };
    ra.currentCommandBuffer = [](void* opaque) {};
    m_player->setRenderAPI(&ra, this);

Metal

    MetalRenderAPI ra{};
    ra.opaque = this;
    ra.currentCommand = [](const void** enc, const void** cmdBuf, const void *opaque) {
        *enc = ;
        *cmdBuf = ;
    };
    ra.device = ;
    ra.cmdQueue = ;
    ra.colorFormat =;        // MUST match render pass
    ra.depthStencilFormat =; // MUST match render pass
    player.setRenderAPI(&ra, this);

Examples:

Render On a Platform Surface

MDK use another project UGS to create and drive a rendering loop for a platform native surface. Call updateNativeSurface() with such a surface value when surface changes, or a null value to create internally. A null value also works for X11 Window or wl_egl_window* or gbm_surface* on linux, EGL_DISPMANX_WINDOW_T* on raspberry pi(legacy driver).

No need to call setRenderCallback() and renderVideo().

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members is enough. For example a D3D11RenderAPI requires context and rtv member can be null. But you can also set RenderAPI Render Context Creation Options fields(see RenderAPI.h comments). And setRenderAPI() MUST be called before the first Player.updateNativeSurface()

    D3D11RenderAPI ra;
    player.setRenderAPI(&ra);
    // ...
    player.updateNativeSurface(....);

Examples:

Render On a Surface Created Internally

Use Player.createSurface() to create variant kinds of surface supported by current OS.

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members MUST be set via setRenderAPI() before createSurface()

examples:

⚠️ **GitHub.com Fallback** ⚠️