Cross compilation

Cross compilation is a great tool of getting a windows application running on the cluster, as UE will semi-automatically seek out any modules that might be needed.

The important things to consider here:

1. Use Vulkan as a rendering backend on the PC to make sure that your project retains compatability
2. Make sure that the vulkan libraries can be found (these should be preloaded with the software stage)
3. Certain modules might hide or alter the environment, which might cause UE to not find a vulkan device
4. If using custom plugins that might rely on binary third party libraries, use module environments first (through path search)

How to setup Unreal Engine on a Compute Cluster

We are focussing this introduction on the Jülich Supercomputing Cluster JURECA or JUWELS, but theoretically this should work on other similarly built supercomputers.

Beware that this is necessarily something that you have to play around with. When I started building UE on the supercomputer, I cause trouble with the system. This document is there so that others can avoid some pitfalls you might encounter. I highlighted some of this in this years course on Unreal Engine for Science already.

NuGet package manager

You might encounter TSL certificate errors if you run the ./GenerateProjectFiles.sh script on a cluster. To avoid this issue, add this statement to the .bashrc or the slurm script:

export SSL_CERT_DIR="/etc/ssl/certs/"

Update on Unreal Engine 5

On our setup, for unknown reasons, the maximum number of file handles as defined in FUnixPlatformFile does not suffice on our system and it will run into the yield so much that I had no patience to let it run. I set the number to 4000 and the building works. You might not want to change this in certain circumstances, but as I am building in shared memory, the number of file handles there is not really an issue.

How to build UE

mkdir /dev/shm/${USER} cd /dev/shm/${USER}

Here you might choose to either use ssh or https with a token. This token you can generate in github under Settings -> Developer Settings -> Personal Access Tokens. You don't need to generate a token with any write rights here, just a simple read-only token would suffice.

git clone --single-branch --depth 1 -b UE_VERSION https://github.com/EpicGames/UnrealEngine.git

I usually do a slightly stricter version of the shallow clone as it helps reduce bloat for projects that are only used. Alternatively, you can do this process by using an Unreal Engine container if that is possible on your platform and workflow (https://unrealcontainers.com/).

cd UnrealEngine/

./Setup.sh

./GenerateProjectFiles.sh

This will take a while. Unreal pulls dependencies and everything that could be needed. The next step is to actually build the applications that are needed. Beware that at this point, you might not actually have control over how many cores are being used. In the folder Engine/Saved/UnrealBuildTool/ there are two xml files: BuildConfiguration.xml and BuildConfiguration.Schema.xml.

This is where you can, with the MaxParallelActions tag, restrict the amount of concurrent operations taken by UE. This is especially important on login nodes, for example, where you are restricted as a user w.r.t. the amount of cores you can use. This prevents UE from trying to spawn more processes (because logical cores are detectable still) than are allocated to you. There are other options, and for the UE build itself, my setup is:

<?xml version="1.0" encoding="utf-8" ?>
<Configuration xmlns="https://www.unrealengine.com/BuildConfiguration">
  <BuildConfiguration>
    <MaxParallelActions>20</MaxParallelActions>
    <bUseIncrementalLinking>true</bUseIncrementalLinking>
    <bEnableThreadSanitizer>true</bEnableThreadSanitizer>
    <bUseUnityBuild>true</bUseUnityBuild>
    <bDisableDebugInfo>true</bDisableDebugInfo>
  </BuildConfiguration>
</Configuration>

(The UE build on /dev/shm is possible on the login node, when you restrict the parallel actions. UBT is not very capable of recognizing how many cores it should use and might stall when trying to access more resources than is permitted. You can omit the MaxParallelActions entry of needed.)

Next important issue is the fact that you do not put -jN after the make command, as UBT will automatically assign processes according to the above mentioned configuration.

After the build has run through, I personally zip the engine into my project folder. You might choose to do differently, or package it in a container. We are aiming at making this project cloud-ready, but focus on HPC machines for the time being.

Make sure that your build configuration is correct. If it isn't, a yellow warning text will be printed at the start of each build call to UE.

Build your project

Using the build sanitizer does not work for project builds. Nevertheless, above configuration is still valid. This will package your project into Saved/StagedBuilds/Linux/ and you will be able to run the projects from a single content file that contains all assets. This is especially important for HPC systems. I still generally advise to develop on windows and build on linux. You can cross-compile, but the general reason why I actively build on the HPC machines is the fact that I might use modules in my packaged game that are important, while also making sure that things like Pixel Streaming operate as expected.

cd /dev/shm/${USER}/UnrealEngine/Engine/Build/BatchFiles/

./RunUAT.sh BuildCookRun -project='/dev/shm/${USER}/project/${PROJECTNAME}.uproject' -platform=Linux -nocompileeditor -clientconfig=Development -build -cook -stage -pak

Synavis cluster environment

You need the nv-codec-headers in the $PATH variable when using H264/H265 codec for encoding. Otherwise VP9 should run out of the box.

The Vulkan libraries must run on the cluster for UE to properly run with all modules. Vulkan-SDK can be provided separately, though the base driver should be accessible through the GPU driver.