Flux - Nerogar/OneTrainer GitHub Wiki

This page is a work in progress as information is learned about Flux.

Flux is a DiT Transformer flowmatching model that has high learning potential but it is a very large and also slow model to work with.

Model Details:

  • Like SD3, a hugginfgace key or a local copy of the diffusers model is needed. OneTrainer now has an option for you to put your HF token in the GUI.
    • AIO (All in One) safetensor models will work.
      • Tested with helheimFlux_v10FP8AIO.safetensors and helheimFlux_v10FP16AIO.safetensors
      • NF4 AIO models will not work (not supported by diffusers)
      • Turbo model safetensors will likely not work and are likely not trainable with OT regardless due to how a Turbo model is made
    • HuggingFace links of finetuned models can work, but the repo must be in diffusers format.
  • Completely De-distilled Flux will not work, as the OneTrainer workflow expects the guidance variable.
  • The standard Flux safetensors file on the Black Forest Labs Hugging Face repo is just the transformer and the VAE, and does not include the text encoders.
  • BFL (Black Forest Labs) has not given all of the details for the model, so some items are still a black box. However, it is important to note that higher resolutions required a time shift. OneTrainer has an option to automatically shift the time step based upon training resolution specifically for Flux.

Limitations:

  • Embeddings do not likely work, due to the nature of T5.
    • Update, Nero has refactored embeddings and output embeddings will now work with Flux.
    • These output embeddings are however only usable in OneTrainer. Use in other software will likely require the OMI format to be finalized first.
  • Some Lora formats (Full Dora) will not work in all generation software. Forge, with a full dora, is known to produce a purple output.
  • FLEX is not currently supported, as it is not quite a Flux.1 dev based model.
  • Flux Schnell is not supported, nor has there been any plans to support it. It is better to use the dedicated Flux trainers if you want to work with this model (Flux Gym, AI Toolkit)

Current Information:

  • Lora is currently the only recommended training. A finetune will require you to eventually train through the distillation of the model.
  • FP8 is the minimum recommended precision to not have artifacts.
  • NF4 precision allows Flux to be used with lower VRAM cards, but it should be noted that a grid pattern can be very visible at this precision level.
  • The OneTrianer Lora can be used in Comfy in the standard Lora Loader.
  • Flux has a robust architecture. It is possible to train a LoRa at 512 or 768 and generate at 1024 with minimal loss in quality.
  • It is possible to train a Flux Lora on 12GB. 8GB cards are likely too close to the limit, and would require windows to be using a different GPU for VRAM usage as Windows can quite a bit of VRAM.