HDR Display Support - JuanDiegoMontoya/Frogfood GitHub Wiki
WIP. PROBABLY CONTAINS MISINFO. PROCEED AT YOUR OWN RISK.
Frogfood supports wide color gamuts and rendering to HDR displays. An HDR display is a display that supports a color standard with a wider gamut and higher peak luminance than sRGB. This enables us to literally render scenes with a higher dynamic range and more color.
Below are two images of the same scene rendered in Frogfood. The first is HDR and can only be properly viewed if your OS and browser are configured, while the second is SDR.
- Windows 10: go to Display Settings and enable HDR on your display.
- Firefox: does not support HDR. Use another browser.
- Chrome and Edge: the default settings should support HDR. If not, going to
chrome://flags(oredge://flagsfor Edge) and setting "Force color profile" to HDR10 or scRGB linear may help.
If your display and browser are correctly configured, the first image should be brighter than the second. Otherwise, it may look darker. This site is useful for checking.
These pictures were captured with Windows Game Bar (one of the few tools that can capture HDR screenshots) and converted to PNG with jxr_to_png.
Tutorial
It's critical to have a crystal clear picture of the color spaces and encodings involved in our rendering pipeline before we try adding HDR output.
This presentation by Rod Bogart explains Unreal Engine 5's color pipeline. All of the concepts from the presentation can be applied to a custom engine.
Whether you know it or not, your engine most likely renders in the RGB color space described by the BT 709 standard. This is because all of the color values going into the renderer are encoded in this space. Since the swapchain is also using this format, the only thing you ever needed to be conscious of is applying the inverse of the sRGB electro-optical transfer function (EOTF) to post-tonemap colors before outputting to the display. This process is commonly known as gamma correction.
The other thing we needed to be careful of is ensuring that texture containing color values (e.g. material albedo) use one of the _SRGB image formats. This tells the API that the data stored in the image is using a nonlinear encoding that must be decoded (by using the sRGB EOTF) when sampling.
This is all a dance to ensure our color values use the most storage-efficient encoding when being stored in a texture or file, while being compatible with math (lighting, blending, and filtering) in our shader.
WTF is an EOTF, IEOTF, or OETF?
In the context of graphics programming, an EOTF can be thought of as the function the display uses to decode your signal which, mind you, is literally compressed up until this point. The compression makes use of the fact that humans perceive differences in low light better than differences in bright light by assigning more bits of the signal to darker areas.
The inverse of the EOTF (the IEOTF) is the function we use to compress our signal prior to sending it to the display. Not using the right IEOTF for your surface's format means your renderer is broken.
This shader shows, in the top row, a gradient without first applying the IEOTF. The middle of the gradient is approximately 50% as perceptually bright as the right end. Despite that, it's only emitting about 22% as much light. In contrast, the gradient on the bottom is radiometrically linear in that the middle is 50% as much light as the right end.
The OETF is sometimes confused to be equivalent to the IEOTF, but they are not the same (as can be confirmed by the fact that the OOTF- the combination of the EOTF and OETF- is not just 1 for every color space where both are defined). As far as I know, you should not worry about this one. Any mention of it here or in my code is almost certainly referring to the IEOTF.
Swapchain
Before creating a swapchain, query the formats supported by your window surface.
auto surfaceFormatCount = uint32_t{};
vkGetPhysicalDeviceSurfaceFormatsKHR(physicalDevice, surface_, &surfaceFormatCount, nullptr);
auto availableSurfaceFormats = std::vector<VkSurfaceFormatKHR>(surfaceFormatCount);
vkGetPhysicalDeviceSurfaceFormatsKHR(physicalDevice, surface_, &surfaceFormatCount, availableSurfaceFormats.data());
VkSurfaceFormatKHR encodes not only the format of the image, but its color space. The Vulkan spec has some useful documentation about the different color spaces.
The Khronos Data Format Specification contains a reference for the various transfer functions mentioned.
In addition to several SDR (sRGB) surface formats, my system supports EXTENDED_SRGB_LINEAR and HDR10_ST2084. You can check this in the Vulkan Caps Viewer.
After enumerating the available VkSurfaceFormats, create a swapchain with the format. Your app must remember what was used here in order to encode the final image correctly.
Rendering
Changing Color Spaces
Once we have the sRGB color pipeline squared away, we can now render in another color space.
"But hold on!", you exclaim. "Why render in another color space?"
Because sRGB has a small gamut. In other words, all the colors that can be expressed as an sRGB triplet are a fraction of what humans with normal color vision can see. Another color space must be used in our renderer if we want to take advantage of the extra colors offered by our HDR displays.
Note that trying to render wide color gamut (WCG) content to an insufficiently wide color space may lead to negative color components being generated.
Frogfood uses either Rec. 709 (sRGB) or Rec. 2020 as its rendering color space. This can be changed at runtime. The image below (from Wikipedia) gives us a sense of the capabilities of some common color spaces.
Since the transformation to a different RGB color space is a linear function, it can be encoded in a 3x3 matrix. All of the functions for converting between color spaces and applying transfer functions can be found in Color.h.glsl.
I used this website to generate the matrices in the following code.
mat3 color_make_sRGB_to_XYZ_matrix()
{
// Transposed to make transcription easier.
return transpose(mat3(0.4123908, 0.3575843, 0.1804808,
0.2126390, 0.7151687, 0.0721923,
0.0193308, 0.1191948, 0.9505322));
}
mat3 color_make_XYZ_to_BT2020_matrix()
{
return transpose(mat3(1.7166512, -0.3556708, -0.2533663,
-0.6666844, 1.6164812, 0.0157685,
0.0176399, -0.0427706, 0.9421031));
}
mat3 color_make_sRGB_to_BT2020_matrix()
{
return color_make_XYZ_to_BT2020_matrix() * color_make_sRGB_to_XYZ_matrix();
}
vec3 color_convert_sRGB_to_BT2020(vec3 srgb_linear)
{
return color_make_sRGB_to_BT2020_matrix() * srgb_linear;
}
Note that XYZ is referring to the CIEXYZ color space, not a generic one. Most color spaces are defined in terms of this one, making it useful as an intermediate color space when doing color space transformations.
With these helpers, all we have to do is use color_convert_sRGB_to_BT2020 on every sRGB input. Inputs that are already in BT.2020 can be left unchanged. There is another helper function that is used to simplify these conversions:
vec3 color_convert_src_to_dst(vec3 color_src, uint src_color_space, uint dst_color_space);
Finally, we perform the conversions in ShadeDeferredPbr.frag.glsl.
We convert all textures to the internal (shading) color space:
const vec3 albedo_internal = color_convert_src_to_dst(texelFetch(Fvog_texture2D(gAlbedoIndex), fragCoord.xy, 0).rgb,
COLOR_SPACE_sRGB_LINEAR,
shadingUniforms.shadingInternalColorSpace);
const vec3 emission_internal = color_convert_src_to_dst(texelFetch(Fvog_texture2D(gEmissionIndex), fragCoord.xy, 0).rgb,
COLOR_SPACE_sRGB_LINEAR,
shadingUniforms.shadingInternalColorSpace);
and all lights:
const vec3 lightColor_internal = color_convert_src_to_dst(light.color,
light.colorSpace,
internalColorSpace);
As you may have noticed, Frogfood assumes all inputs are linear sRGB except for lights. This is an easy way to have some WCG content to test in your renderer.
With our inputs in the same color space, we can do our fancy PBR calculations to generate the raw scene image.
Tonemapping
Tonemapping for HDR is a bit tricky as it's no longer about simply cramming the range [0, inf] into [0, 1]. We are now supposed to think of the physical luminance we want the display to emit while taking into consideration its capabilities: max full frame luminance, max 10% luminance, and more. That's right, you aren't even guaranteed a single value for max brightness! hgig.org has a great presentation on the topic.
For HDR, Frogfood uses the Gran Turismo tonemapper. This one was chosen due to its configurable peak luminance, among other parameters. Some of its developers gave an amazing presentation detailing it and other information about HDR.
Anyway, once you have an HDR-aware tonemapper, call it using the scene color and our display's peak brightness as the input.
const vec3 tonemappedColor = GTMapper(hdrColor, uniforms.gt, uniforms.maxDisplayNits);
With our tonemapped color that is in [0, maxDisplayNits], all that's left is to convert it into the color space and encoding that our swapchain requires, then dither if desired.
const vec3 outputColor = ConvertShadingToTonemapOutputColorSpace(tonemappedColor, uniforms.shadingInternalColorSpace, uniforms.tonemapOutputColorSpace);
vec3 ditheredColor = outputColor;
if (bool(uniforms.enableDithering))
{
// Our dither should be aware of the bit depth of the backbuffer- do not assume 8 bits.
ditheredColor = apply_dither(outputColor, uv, uniforms.quantizeBits);
}
ConvertShadingToTonemapOutputColorSpace is a doozy. It supports converting between any of the internal color spaces and any of the output (swapchain) color spaces supported by Frogfood. Since only two internal color spaces are supported, the function is relatively small.
vec3 ConvertShadingToTonemapOutputColorSpace(vec3 color_in, uint in_color_space, uint out_color_space)
{
switch (in_color_space)
{
case COLOR_SPACE_sRGB_LINEAR:
switch (out_color_space)
{
case COLOR_SPACE_sRGB_LINEAR: return color_in;
case COLOR_SPACE_sRGB_NONLINEAR: return color_sRGB_OETF(color_in);
case COLOR_SPACE_scRGB_LINEAR: return color_in * uniforms.maxDisplayNits / 80.0;
case COLOR_SPACE_BT2020_LINEAR: return color_convert_sRGB_to_BT2020(color_in);
case COLOR_SPACE_HDR10_ST2084: return color_PQ_OETF(color_convert_sRGB_to_BT2020(color_in) * uniforms.maxDisplayNits / 10000.0);
}
break;
case COLOR_SPACE_BT2020_LINEAR:
switch (out_color_space)
{
case COLOR_SPACE_sRGB_LINEAR: return color_convert_BT2020_to_sRGB(color_in);
case COLOR_SPACE_sRGB_NONLINEAR: return color_sRGB_OETF(color_convert_BT2020_to_sRGB(color_in));
case COLOR_SPACE_scRGB_LINEAR: return color_convert_BT2020_to_sRGB(color_in) * uniforms.maxDisplayNits / 80.0;
case COLOR_SPACE_BT2020_LINEAR: return color_in;
case COLOR_SPACE_HDR10_ST2084: return color_PQ_OETF(color_in * uniforms.maxDisplayNits / 10000.0);
}
break;
}
UNREACHABLE;
return color_in;
}
imgui.frag.glsl contains a more complete function for performing arbitrary conversions. This allows us to correctly render images of any color space with ImGui, which is quite important as Frogfood draws the scene to an ImGui window.
Calibration
Normally, the tonemapper would be configured with HDR metadata queried by DXGI or other OS-level APIs. Frogfood instead implements a "calibration checkerboard". The tool consists of two components: a brightness slider and a grid.
The user's objective is to increase the brightness until the dark squares and the bright squares are indistinguishable. In a world where HDR displays simply clipped if content went over the peak brightness, this tool would be almost perfect. However, many displays perform their own tonemapping to smoothly remap [0, 10000] nits to [0, maxDisplayNits], rendering our effort sub-perfect.
Implementation
The checkerboard is implemented as a simple compute shader that draws to a 2x2 texture, which is then magnified with a repeat sampler.
void main()
{
const ivec2 gid = ivec2(gl_GlobalInvocationID.xy);
if (any(greaterThanEqual(gid, imageSize(Fvog_image2D(outputImageIndex)))))
{
return;
}
// HDR: Emit 10000 nits for 50% checkerboard. The other 50% should be displayTargetNits.
// If they approximately match to a viewer, the max display brightness has been found.
// Note 1: while the peak brightness of most monitors is <<10000 nits, some smoothly approach that peak
// (tonemap) as the input reaches 10000, so any calibration will be somewhat wrong on those monitors.
// Note 2: the test image should take up a small percentage of the screen (2-10%) as the peak brightness
// generally cannot be sustained on the whole display. Therefore, scene content should aim to have peak
// brightness cover only a small portion of the screen as well.
vec3 color = vec3(1); // 10k nits
if ((gid.x + gid.y) % 2 == 0)
{
color = color_PQ_OETF(vec3(displayTargetNits / 10000.0));
}
imageStore(Fvog_image2D(outputImageIndex), gid, vec4(color, 1.0));
}
Mixing SDR and HDR content
TODO
Gamut Compression
While WCG content is cool, not everyone's display is up for the task of rendering it. Displays that don't support 100% of the standard that our content is authored in will show clipped colors that Just Look Wrong.
TODO
Debugging
TODO
Surface Format Reference
This section is split into separate parts for VkFormat and VkColorSpaces, since they can be thought of independently.
VkFormat
- Any
_UNORMformat implies the input must be in [0, 1] and no automatic munging of your color values will be performed when rendering to the image. - Any
_SRGBformat implies the input must be in [0, 1] and the sRGB linear-to-nonlinear transfer function (AKA the OETF) will be applied when rendering to the image. These formats usually correspond to theSRGB_NONLINEARcolor space, but in MoltenVK I've seen it paired with literally every color space. For non-sRGB color spaces, that means you'd have to encode the colors as dictated by the color space, then apply an additional sRGB EOTF on top in anticipation of the OETF that will be automatically applied. - Any
_SFLOATformat implies the input must be in [-inf, inf] and no conversion will be performed on write.
VkColorSpace
SRGB_NONLINEAR: likely what you have been using before. The input is expected to be encoded in the BT.709 color space with the sRGB nonlinear transfer function. The conversion from linear to nonlinear is required if the image format isn't_SRGB, and looks like:
vec3 color_sRGB_OETF(vec3 srgb_linear)
{
bvec3 cutoff = lessThan(srgb_linear, vec3(0.0031308));
vec3 higher = vec3(1.055) * pow(srgb_linear, vec3(1.0 / 2.4)) - vec3(0.055);
vec3 lower = srgb_linear * vec3(12.92);
return mix(higher, lower, cutoff);
}
HDR10_ST2084: a standard that specifies a 10k nit peak luminance and uses the BT.2020 color space._ST2084refers to the transfer function, which is a PQ curve that looks like:
// Input should be in [0, 10000] nits. Usage: color_PQ_OETF(x * max_nits / 10000.0);
float color_PQ_OETF(float x)
{
const float m1 = 0.1593017578125;
const float m2 = 78.84375;
const float c1 = 0.8359375;
const float c2 = 18.8515625;
const float c3 = 18.6875;
float ym = pow(x, m1);
return pow((c1 + c2 * ym) / (1.0 + c3 * ym), m2);
}
EXTENDED_SRGB_LINEAR: also known as scRGB, this color space is literally an extension of sRGB that supports negative color components to embiggen the gamut. Like sRGB, a value of 1 corresponds to 80 nits. Unlike sRGB, values above 1 correspond to a linear increase in brightness. Most SDR displays don't respect that part of the spec, so sRGB content displayed as scRGB may look dimmer than you're used to. Since this color space is linear, no linear-to-nonlinear transfer function is needed. However, you still need to account for the display's peak brightness:
return color_in * uniforms.maxDisplayNits / 80.0;
BT2020_LINEAR: this one behaves weirdly on my system, and I don't know why. Maybe it's a driver bug? Anyway, the spec says it's BT.2020 encoded using a linear transfer function. Since it's paired with a_UNORMformat on my system, it seems to only be for wide color gamut output, but not "HDR" in the sense of increased dynamic range. Despite that, my system seems to treat it exactly the same asSRGB_NONLINEARand attempting to output BT.2020 colors just leads to a darkened, oversaturated appearance.
More HDR-SDR Comparison Pictures
Try to ignore the broken foliage in the next two pics (the path tracer doesn't yet support masked geometry).