Training Flux LoRA Models with FluxGym, Zluda, and ROCm on Windows - likelovewant/ROCmLibs-for-gfx1103-AMD780M-APU GitHub Wiki

Setting up FluxGym for ROCm and Zluda

This guide explains how to configure FluxGym for training LoRA models using ROCm and the Zluda library on windows.

Prerequisites:

FluxGym: Install FluxGym following the instructions on its GitHub page: https://github.com/cocktailpeanut/fluxgym
ROCm: Ensure you have a compatible ROCm installation (version 5.7 or 6.1).

Steps:

Environment Activation: Activate your FluxGym environment using env/Scripts/activate.
Torch Installation: Install PyTorch with the correct CUDA support for ROCm:
```
pip install torch==2.3.1 torchvision --index-url https://download.pytorch.org/whl/cu118
```
Note: You can experiment with different PyTorch versions, but make sure to use cu118 compatible with ROCm. Zluda relies on this specific version.
ROCm Library Matching: Replace the rocmlibs in your HIPS DK 5.7 or 6.1 installation with the corresponding libraries from your chosen ROCm version.
Zluda Integration:
- Download the appropriate Zluda release based on your ROCm version (HIP SDK 5.7.1 or 6.1.2) from: https://github.com/lshqqytiger/ZLUDA/releases.
- Rename the essential CUDA libraries according to Zluda's naming convention:
  - cublas.dll becomes cublas64_11.dll
  - cusparse.dll becomes cusparse64_11.dll
  - nvrtc.dll becomes nvrtc64_112_0.dll
- If using AMDgpu Forge or SD.Next: Obtain the required Zluda files from the default .zluda folder within your installation directory and place them in fluxgym\env\Lib\site-packages\torch\lib, overwriting the existing files.

Disable Cudnn and XFormers:

Open flux_train_network.py located at fluxgym\sd-scripts.

On line 7, where you have import torch, add the following code:

import torch
torch.backends.cudnn.enabled = False
torch.backends.cuda.enable_flash_sdp(False)
torch.backends.cuda.enable_math_sdp(True)
torch.backends.cuda.enable_mem_efficient_sdp(False)
if hasattr(torch.backends.cuda, "enable_cudnn_sdp"):
    torch.backends.cuda.enable_cudnn_sdp(False)

These modifications disable Cudnn and XFormers, as Zluda currently does not support them.

Training:

Now you can run FluxGym to train your LoRA model. Remember that these steps apply to other PyTorch CUDA programs as well. You can adjust the configuration based on your specific project requirements.