Objectives

In this tutorial, we will learn how to create a simple overlay display - drawn with Direct3D 11 - that can be shown in any OpenXR application using Direct3D 11, Direct3D 12 or Vulkan.

Prerequisites: Tutorial 1: Your first OpenXR API layer.

The code for this tutorial can be found in the branch: examples/overlay-basic-d3d11

What it will look like:

This is very basic... but it is your first overlay! In the next tutorials, we will learn how to add interactions and how to import windows such as WinForms.

Before we start

Overlays in OpenXR

We will briefly review some fundamental concepts of OpenXR that we will employ to create our overlay.

In OpenXR, the simplest way to draw a 2D overlay is by submitting a "quad layer" (XrCompositionLayerQuad) with the frame.

The XrCompositionLayerQuad layer is useful for user interface elements or 2D content rendered into the virtual world. The layer’s XrSwapchainSubImage::swapchain image is applied to a quad in the virtual world space. Only front face of the quad surface is visible; the back face is not visible and must not be drawn by the runtime. A quad layer has no thickness; it is a two-dimensional object positioned and oriented in 3D space. The position of a quad refers to the center of the quad within the given XrSpace. The orientation of the quad refers to the orientation of the normal vector from the front face. The size of a quad refers to the quad’s size in the x-y plane of the given XrSpace’s coordinate system.

The content of the quad layer must be drawn into a swapchain, which can be done once (for static content), every frame (for animated content) or when necessary (for content that is updated on event only). A swapchain can only be used through the graphics API that the OpenXR session is currently using. It becomes challenging for an API layer to implement its drawing code for all possible graphics API... For this reason, we provide graphics utilities (see below).

The placement of a quad layer is relative to the space specified for the quad layer. This space will typically be a reference space (such as the view space for head-locked overlays, or the local space for world-locked content), but it could also be an action space (for example: to draw the content relative the user's hands).

The API layer template's graphics utilities

Typically, an API layer developer would use the basic functionality of the OpenXR API layer template as described in the first tutorial: override selected OpenXR functions, then use the OpenXR API directly to implement new functionality. However, for certain features, such as rendering (for graphics overlay presented here) and inputs (presented in the next tutorial), comes an important challenge: manipulating graphics resources with the OpenXR API in an API layer is tied to the host application. For example, if the application running below the API layer is using Direct3D 11 for rendering, this means the API layer must manipulate and submit graphics resources using Direct3D 11 as well. For an API layer developer to create an API layer that will work on Direct3D 11, Direct3D 12 and Vulkan applications, would normally require to write the same drawing code with each of those graphics API...

Fortunately, the OpenXR API layer template comes with a graphics utility library that provides an abstraction of the key concepts needed to render with OpenXR and will let you write your graphics code once with Direct3D 11, and still support when host applications are using Direct3D 12 and (soon) Vulkan.

The rest of this tutorial will heavily use the graphics utility library!

Step by step

We start by declaring in the framework/layer_apis.py script the OpenXR functions we will be changing the behavior of. We will keep our implementation of xrGetSystem() from the previous tutorial, and we will add xrEndFrame(), the key override where we will add our quad layer to form our overlay.

# The list of OpenXR functions our layer will override.
override_functions = [
    "xrGetSystem",
    "xrEndFrame"
]

We then declare the OpenXR functions we will be using from the OpenXR runtime. For our simple overlay, we will only need xrCreateReferenceSpace()/xrDestroySpace() in order to create and manage a reference space to anchor the overlay to. We keep the other functions that we used in the previous tutorial as well for querying instance and system properties for logging purposes.

# The list of OpenXR functions our layer will use from the runtime.
# Might repeat entries from override_functions above.
requested_functions = [
    "xrGetInstanceProperties",
    "xrGetSystemProperties",
    "xrCreateReferenceSpace",
    "xrDestroySpace"
]

We can now move on to adding the code to initialize and maintain the graphics::ICompositionFrameworkFactory that we will use to query the graphics::ICompositionFramework objects necessary to manage swapchains and to perform interoperation between Direct3D 11 and whichever graphics API the OpenXR application might be using.

XrResult xrCreateInstance(const XrInstanceCreateInfo* createInfo) override {
    [...]

    // Initialize the composition framework.
    m_compositionFrameworkFactory = graphics::createCompositionFrameworkFactory(
        *createInfo, GetXrInstance(), m_xrGetInstanceProcAddr, graphics::CompositionApi::D3D11);

    return XR_SUCCESS;
}

The snippet above shows creating an instance of the ICompositionFrameworkFactory object and tying it to the OpenXR XrInstance that our API layer is hooking into. We do this in the xrCreateInstance() hook, which is the entry point for our API layer hooks.

Note that we declare our intention to use Direct3D 11 for composition (drawing) of our overlay.

XrResult xrGetInstanceProcAddr(XrInstance instance, const char* name, PFN_xrVoidFunction* function) override {
    [...]

    // Required to call this method for housekeeping.
    if (m_compositionFrameworkFactory) {
        m_compositionFrameworkFactory->xrGetInstanceProcAddr_post(instance, name, function);
    }

    [...]

One of the key requirements of the ICompositionFrameworkFactory object is to be able to intercept of the application calls on its own. For the reason, we must modify the implementation of xrGetInstanceProcAddr() as shown in the code above to invoke the xrGetInstanceProcAddr_post() method after the upstream call (real call) to xrGetInstanceProcAddr() was performed.

Now that we are done with the formalities of the ICompositionFrameworkFactory object, let us move on to the end-goal part of this tutorial: our overlay. We will need to initialize a swapchain for our quad layer and a reference XrSpace to place it in the user's view.

XrResult xrEndFrame(XrSession session, const XrFrameEndInfo* frameEndInfo) override {
    [...]

    graphics::ICompositionFramework* composition = m_compositionFrameworkFactory->getCompositionFramework(session);
    CompositionData* compositionData = composition->getSessionData<CompositionData>();

We begin with the code above in the implementation of xrEndFrame(): first we must query the ICompositionFramework for the OpenXR session. This object has the methods that will let us manage swapchains to draw our overlay. Next, we retrieve the state that we stored for this session using the ICompositionFramework::getSessionData() template method. The data stored in the CompositionData object has the same lifetime as the session, and therefore we can conveniently use that class to store the resources needed for our overlay.

    // First time: initialize the resources for the session.
    if (!compositionData) {
        // Allocate storage for the state.
        composition->setSessionData(std::move(std::make_unique<CompositionData>(*this)));
        compositionData = composition->getSessionData<CompositionData>();

        // Create a swapchain for the overlay.
        XrSwapchainCreateInfo overlaySwapchainInfo{XR_TYPE_SWAPCHAIN_CREATE_INFO};
        overlaySwapchainInfo.usageFlags = XR_SWAPCHAIN_USAGE_COLOR_ATTACHMENT_BIT;
        overlaySwapchainInfo.arraySize = 1;
        overlaySwapchainInfo.width = 512;
        overlaySwapchainInfo.height = 512;
        overlaySwapchainInfo.format = composition->getPreferredSwapchainFormatOnApplicationDevice(overlaySwapchainInfo.usageFlags);
        overlaySwapchainInfo.mipCount = overlaySwapchainInfo.sampleCount = overlaySwapchainInfo.faceCount = 1;
        compositionData->overlaySwapchain = composition->createSwapchain(overlaySwapchainInfo, graphics::SwapchainMode::Write | graphics::SwapchainMode::Submit);

        // Create a head-locked reference space.
        XrReferenceSpaceCreateInfo createViewSpaceInfo{XR_TYPE_REFERENCE_SPACE_CREATE_INFO};
        createViewSpaceInfo.referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW;
        createViewSpaceInfo.poseInReferenceSpace = Pose::Identity();
        CHECK_XRCMD(OpenXrApi::xrCreateReferenceSpace(session, &createViewSpaceInfo, &compositionData->viewSpace));
    }
    
    [...]

Before drawing our overlay for the first time, we create, as shown above, a swapchain that can be both drawn into (XR_SWAPCHAIN_USAGE_COLOR_ATTACHMENT_BIT and SwapchainMode::Write are used to declare this intent) and submitted to the OpenXR runtime (SwapchainMode::Submit). By using the ICompositionFramework object to create the swapchain, we are setting up the swapchain interoperability that allows us to use Direct3D 11 to draw the overlay, regardless of the graphics API that the underlying application uses. With the graphics::ISwapchain object that we now have, we only need to take care of one thing only: drawing the overlay. Every other implementation detail will be taken care of by the underlying utilities framework.

For this tutorial, we will place our overlay right in front of the user, and make it follow the movement of the user so that the overlay is always visible. This is done by using the reference space XR_REFERENCE_SPACE_TYPE_VIEW. You may change this to XR_REFERENCE_SPACE_TYPE_LOCAL to make the overlay world-locked, but beware that without giving the user the ability to move the overlay around (which we will learn in a future tutorial), the placement might be inconvenient and bothersome.

It is now time to do our drawing. Still inside the xrEndFrame() hook, we will need to follow a series of steps to interact with the swapchain, and gather the necessary resources from the utilities framework.

    graphics::ISwapchainImage* const acquiredImage = compositionData->overlaySwapchain->acquireImage();
    {
        ID3D11Device* const device = composition->getCompositionDevice()->getNativeDevice<graphics::D3D11>();
        ID3D11DeviceContext* const context = composition->getCompositionDevice()->getNativeContext<graphics::D3D11>();
        ID3D11Texture2D* const surface = acquiredImage->getTextureForWrite()->getNativeTexture<graphics::D3D11>();

These 4 lines above will give us everything we need to render our overlay to the surface that we will submit to OpenXR. We can then (finally) do some drawing! Any Direct3D construct may be used here, and we only show some simple code to give an idea of how to use our rendering context.

        // Create an ephemeral render target view for the drawing.
        ComPtr<ID3D11RenderTargetView> rtv;
        D3D11_RENDER_TARGET_VIEW_DESC rtvDesc{};
        rtvDesc.ViewDimension = D3D11_RTV_DIMENSION_TEXTURE2D;
        rtvDesc.Format = (DXGI_FORMAT)swapchainInfo.format;
        rtvDesc.Texture2D.MipSlice = D3D11CalcSubresource(0, 0, 1);
        CHECK_HRCMD(device->CreateRenderTargetView(surface, &rtvDesc, rtv.ReleaseAndGetAddressOf()));

        // Draw to the surface.
        ComPtr<ID3D11DeviceContext1> context1;
        CHECK_HRCMD(context->QueryInterface(context1.ReleaseAndGetAddressOf()));
        context1->OMSetRenderTargets(1, rtv.GetAddressOf(), nullptr);

        const float background[4] = {0.0f, 0.0f, 0.0f, 0.5f};
        const float red[4] = {1.0f, 0.0f, 0.0f, 1.0f};
        const float green[4] = {1.0f, 0.0f, 0.0f, 1.0f};
        context1->ClearRenderTargetView(rtv.Get(), background);

        D3D11_RECT rect1;
        rect1.left = 10;
        rect1.top = 10;
        rect1.right = swapchainInfo.width / 2 - 10;
        rect1.bottom = swapchainInfo.height - 10;
        context1->ClearView(rtv.Get(), red, &rect1, 1);

        D3D11_RECT rect2;
        rect2.left = swapchainInfo.width / 2 + 10;
        rect2.top = 10;
        rect2.right = swapchainInfo.width - 10;
        rect2.bottom = swapchainInfo.height - 10;
        context1->ClearView(rtv.Get(), green, &rect2, 1);

Note: for those who wish to not create a new render target view every time, you may use something like SetPrivateData() or SetPrivateDataInterface() to store data in the Direct3D resources for future use.

Note: in this example, we re-draw the content of the quad layer every frame. However, this code could be optimized to only do the drawing when necessary. A swapchain that is referenced in a quad layer will use the content of the most recently "released" image (see below).

Once we are done, there are a few more steps to complete and put the swapchain in a submittable state.

        ID3D11RenderTargetView* nullRTV = nullptr;
        context1->OMSetRenderTargets(1, &nullRTV, nullptr);
    }
    compositionData->overlaySwapchain->releaseImage();
    compositionData->overlaySwapchain->commitLastReleasedImage();

The code above wraps up our drawing, first by clearing references to any bound resources (here, the render target view), then by committing our drawings to the swapchain.

It is now time to tell OpenXR to draw our quad layer with the frame submitted by the application.

    XrCompositionLayerQuad overlay{XR_TYPE_COMPOSITION_LAYER_QUAD};
    overlay.layerFlags = XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT;
    overlay.subImage = compositionData->overlaySwapchain->getSubImage();

    // Place the overlay.
    overlay.eyeVisibility = XR_EYE_VISIBILITY_BOTH;
    overlay.pose = Pose::Translation({0.0f, 0.0f, -1.0f});
    overlay.space = compositionData->viewSpace;
    overlay.size.width = 1.0f;
    overlay.size.height = 1.0f;

    // Append our overlay quad layer.
    std::vector<const XrCompositionLayerBaseHeader*> layers(chainFrameEndInfo.layers, chainFrameEndInfo.layers + chainFrameEndInfo.layerCount);
    layers.push_back(reinterpret_cast<XrCompositionLayerBaseHeader*>(&overlay));

    XrFrameEndInfo chainFrameEndInfo = *frameEndInfo;
    chainFrameEndInfo.layers = layers.data();
    chainFrameEndInfo.layerCount = (uint32_t)layers.size();

    return OpenXrApi::xrEndFrame(session, &chainFrameEndInfo);

In the snippet of code above, we show how to create a copy of the XrFrameEndInfo struct that is writable, so that we can append our quad layer. We fill out a XrCompositionLayerQuad structure to describe its content (using the helper ISwapchain::getSubImage()) and its placement in space, in our case 1 meter in front of the user, and with a size of 1 square meter.

Finally, we pass our new frame descriptor to the real xrEndFrame().

Conclusion

You can now experiment with the code presented in this tutorial, by drawing a more complex overlay, and by playing around with its placement in space.

If you have any feedback on this tutorial or the example code, please file an issue.

Thank you.

Next tutorial: Tutorial 3: Adding motion controller inputs to your overlay.