WIP ‐ Shading introduction ‐ Vanilla, Complex material, PBR - doodlum/skyrim-community-shaders GitHub Wiki

This page aims to intuitively explain how shading works in vanilla Skyrim, what Complex Material adds, and their differences to PBR. It includes the basics of how the light that hits objects is calculated, how textures control the look of objects, and shortly about the meshes' role in this. I will try to keep it as approachable as possible, applicable outside Skyrim where relevant, and include references for further reading. All of this will be informal and dumbed down, to get a deeper understanding I recommend pbr-book.org and Advanced Global Illumination .

Lighting in general

For a very simplified model of lighting, light can be considered to be made up of rays that travel in straight paths at infinite speed. To simplify further, light can be divided into direct and indirect. Direct light hits a surface right after being emitted; this can also be called local illumination. Indirect light, also bounce light or global illumination, is light that arrives at the surface after bouncing around the scene.

(source)

Direct light is simple: we can determine if a surface is lit based on the angle between the surface and the light ray and the distance from the light source. The surface can also be occluded by another object - in Skyrim, some lights render shadow maps to be able to check if a surface is in shadow.

Indirect light is harder, and Skyrim uses an extremely simple way to approximate it. An ambient color is used as a replacement for any light bouncing around in the scene. In some cases Skyrim uses six colors, one for each axis (i.e. one applied to each side of a cube). These colors are set by artists and have no real connection to what is actually in the scene. Ambient occlusion 'occludes' or 'shadows' the ambient light, and is a way to make indirect lighting more believable. Skyrim includes SSAO, which darkens edges and corners to approximate less light getting into those places. ENB and Community Shaders have more advanced ways to improve indirect lighting.

There are three important terms describing how a light can bounce from a surface:

• Diffuse: the ray bounces in a random direction, the surface looks the same no matter where you look from
• Glossy: the ray tends to bounce closer to the mirror-like direction
• Specular: the ray bounces exactly at the same angle mirrored by the normal vector (it's basically a mirror)

(source) (source)

Diffuse and specular reflection are the opposite extremes. Real materials are always somewhere in between.

Shortly about color spaces

Colors and our perception of them is extremely complicated (just look at this article series). There are many color spaces (representations of color) used for various purposes. For now, let's ignore everything except one thing - sRGB or linear space (detailed introduction).

sRGB:

• Represents how we see color - 2x higher value looks 2x brighter
• Doing math with it doesn't make much sense

Linear:

• Represents how light intensity works - 2x higher value is (but does not look) 2x more intense
• Math works correctly

To see color correctly, it needs to be in sRGB, to do calculations correctly, it needs to be in linear. Converting between them is simple but might have been more expensive on old hardware. If you ever see a texture that is really dark and saturated or very bright and desaturated, it might be in the wrong space.

(source)

As an aside: Color data are usually stored in sRGB because it is more efficient to store bits we actually see than those that are good for math. A dds texture being flagged as sRGB signifies that it stores color data, and the gpu automatically converts the color to linear when sampling. Conversely, a linear flagged textures signifies that the data is already linear and no conversion is done. These flags did not exist when the original Skyrim was being made.

Shortly about texture channels

Textures have up to 4 channels - RGBA (red, green, blue, alpha). Each channel can be interpreted as a grayscale image. Grayscale textures are usually saved as R or RGB where all channels have the same value. There can be some exotic combinations of channels, but in Skyrim modding you will see textures with R, RGB, or RGBA channels.

There is no real difference between the channels and sometimes it is convenient to put unrelated data in each channel (channel packing). In normal images, the A channel is used for transparency, so image editors usually treat it differently from the other channels.

Material properties used in shading

A general explanation of properties that can describe a material follows. They don't all work together, some are alternatives to each other or only make sense with certain shading models. The properties can be of a single value describing the whole surface, or a texture where the values differ per pixel. Some of these properties usually have an associated multiplier to give more control to the artist. Some related math and nice pictures.

• Albedo/Base color: The proportion of light reflected by an object. Basically determines how bright the object is and its color in neutral light.
• Diffuse map: Sometimes used interchangeably with Albedo, but usually has static lighting applied to it. This includes baked ambient occlusion or specular highlights.
• Specular color: The tint of specular reflections on the object. Metals have colored reflections, while nonmetallic reflections retain the original colors.
• Specular level: The amount of light a specular reflection gives. Closely related to index of refraction (IOR). Most opaque materials have similar specular levels. Materials that differ are for example: water, snow, ice, various gemstones.
• Roughness: How rough the material is from fully specular to fully diffuse. Rougher = the specular highlight is more spread out. The amount of light reflected stays the same.
• Glossiness: The opposite of roughness. Can be rarely called 'specular power' not as in how powerful the specular is, but mathematically as in value^power - in this definition the higher power makes the specular highlight more concentrated, which is consistent with glossiness. Also named shininess in some parts of Skyrim.
• Metallic: Controls whether the material behaves like a metal. Metals' specular reflections are tinted and they have no diffuse color. In setups using this map, the Albedo serves as diffuse color for nonmetals and specular color for metals.
• Ambient Occlusion: Determines how much ambient light hits the surface. Darker AO = less ambient light. Used to cheaply approximate indirect light.
• Normal map: Determines which direction the surface is pointing in. Adds the impression of geometry detail on a flat surface.
• Environmental mask: Determines how much the environment is reflected. In Skyrim used together with a cubemap representing the environment.

Vanilla Skyrim Lighting

The following information is based on shader code in Community Shaders, which was reverse engineered from Skyrim. The relevant sections: specular calculation, directional diffuse, other lights.

(source)

In short, Skyrim seems to use the Blinn–Phong shading model which is an optimization of the Phong model. It is a simple (by today's standards) model that calculates a diffuse and a specular contribution of a light. It is very performant for a model with specular highlights. But it does not have properties required for realistic lighting, the most important in this context being the conservation of energy, meaning that some materials reflect more or less light than is coming in (relevant article). Some comparisons with a more realistic model are included in this paper by Blinn.

Skyrim uses two textures for default shading (there are others for fancy effects). They are the diffuse and the normal map. In the alpha channel of the normal map is a specular map. Therefore Skyrim can do normal mapped materials with variable highlight intensity (but not highlight size). In the nif (mesh file), several properties can be set for the whole mesh: glossiness, specular strength (intensity multiplier), and specular color.

The diffuse component is calculated by multiplying the light color, the cosine between the light and normal vectors, and the light intensity. This value should be divided by pi which seems to have been "absorbed" into light intensity; you can read an article about that here. Finally, the value is multiplied by the color from the diffuse texture.

The specular component is calculated by taking the cosine of the half and normal vectors to the power of shininess (glossiness). This is multiplied by the specular level from the texture (A channel in the normal map) and the multiplier set in the nif. Note that the specular level is named 'glossiness' in the shader, but it does not behave as glossiness.

To me, there are three main weaknesses in Skyrim that make vanilla shading not ideal for the future:

1. Since the shading model is so simple, it doesn't do desirable effects we see in real life such as fresnel. Metals can only be faked using environment mapping.
2. All the math is done in sRGB, which is bad bad. Multiplying and adding up sRGB values is wrong, and all the colors in the process described are in sRGB in Skyrim. It is this way most likely for performance (to avoid conversion), and some mobile games still do this now. The diffuse textures are seemingly halfway between sRGB and linear space, but for all intents and purposes they are sRGB (the actual data; the dds is flagged as linear). You might actually prefer the look of sRGB lighting in some cases, but the point is that it makes colors set by artist wrongly skewed. This image from Unity shows the error that appears when calculating with sRGB values:

(source)

3. The properties stored in the nifs have weird scaling, and most people don't know about them. As a texture author, unless you set the glossiness and specular level in the nif, you have basically no control over how your texture looks. Nobody ships meshes with their textures unless the meshes are original, and shipping meshes also makes compatibility with mesh fixes and other mods much worse. But without that, you are at the mercy of the properties set in vanilla.

Regarding points 1. and 2., ENB does some effects that improve the situation, but I don't know how it is implemented. The shading model can be adjusted to conserve energy or even replaced for a more complicated model, and calculations can be done in linear space. The main reason Community Shaders doesn't do this (yet, and if it does then not by default) is that it may significantly change (or break) the look of vanilla materials, which is not a goal of Community Shaders.

Parallax (very shortly)

Parallax occlusion mapping simulates the height/displacement of the surface without needing actual geometry. The parallax effect shifts the texture coordinates based on view angle give a 3d effect, but does not otherwise contribute to the shading. As parallax only shifts the texture coordinates, it cannot extend outside the bounds of the actual geometry, but it can at be used to cast small shadows on the same surface.

Vanilla Skyrim does have parallax support, but the shader is broken and needs mods to work correctly. ENB (supposedly) distinguishes parallax from complex parallax, where the difference is in the visual quality of the algorithm. This is sometimes conflated with Complex (parallax) material, which also has the parallax effect, but without using the Skyrim Parallax shader type (which has some limitations).

Environment Mapping

Vanilla Skyrim supports environment mapping from a cubemap. It uses two special textures: the cubemap, and an environment mask. The cubemap texture contains an unwrapped cube (like old skyboxes) which contains the image that is 'reflected', while the mask determines how strong the reflection is. There is also a multiplier for cubemap intensity inside the nifs.

The cubemaps used are either a prerendered scene from Skyrim, or some abstract shapes trying to replicate a colored specular or a more complicated effect such as the rainbow reflections on oil.

(source) (vanilla chrome reflection)

ENB and eventually Community Shaders added the option to use a dynamic cubemap, which is continuously updated using what's rendered with the player camera. In other words, there are no separate render passes, only what the player sees is present. Therefore the environment is not always captured fully or timely, but the effect is relatively cheap. Once again, one has to be careful applying dynamic cubemaps to objects, as the vanilla multipliers may be arbitrary (vanilla dwemer metals would reflect the environment 3x brighter).

Complex Material

The environment mask discussed in the previous section only uses one channel. ENB complex material uses the other channels to provide additional information to the shader. For the texture to be used, the environment map shader type must be set (discussed in the previous section).

Complex material has (source):

• R: environment mask, same as in vanilla,
• G: glossiness, multiplies the vanilla glossiness property,
• B: metallic, makes the diffuse texture control specular color instead of diffuse color,
• A: parallax, same as in the parallax shader type,

Complex material best known for allowing the use of parallax with environment mapping. It is backward compatible in the sense that vanilla does not break when encountering it, instead the additional channels are simply ignored.

Complex material uses the same shading as vanilla, therefore all caveats mentioned previously apply. Although the textures available resemble a PBR setup, the shading is still not physically based and is not in linear space. The use of an environment mask and the glossiness and specular multipliers in the nif makes it hard to control the visual.

PBR (Community Shaders implementation)

TBD