Backup and Save - Nerogar/OneTrainer GitHub Wiki
The Backup Tab
Its important to clarify between a backup and a save:
Backup
- A Backup is for resuming training only, it will not work in inference front-ends.Save
- If you want to test intermediate training epochs/steps, then saves are what you load into inference UI's (or the final output model).
NOTE: Backups and saves are located in the workspace
directory. To resume from your most recent backup see the General tab settings or for a specific Model tab
Backup Focused Settings
Backup after X Y
- This will create a backup after X Epochs/Steps/Second/Minute/Hour/Never/Always. Note that Never and Always are special and will ignore your integer.Backup now
- Pressing this button will tell OneTrainer to do a backup at the next opportunity.Rolling Backup
- A storage saving setting. Enabling this will automatically delete backups as new ones are created.Rolling Backup Count
- The number of backups OneTrainer will keep. Older ones will automatically be deleted if Rolling Backup is enabled.Backup Before Save
- This setting ties backup creation to save creation. This can used in conjunction with your other backup settings, or you can use your save settings to control the backup creation.
Save Focused Settings
Save every X Y
. This will create a save every X Epochs/Steps/Second/Minute/Hour/Never/Always. Note that Never and Always are special and will ignore your integer.Skip First X
: Skip the first X intervals (Epochs/Steps/Second/Minute/Hour). Ex: save every 10 epochs and skip first 100 will start saving after 110 epochs.Save Filename Prefix
. A requested feature to help better sort saves. This will add a prefix to the standard OneTrainer save format.
Notes and Comments
- Changes to this tab will be reflected in a running training. For example if you are on epoch 9, and want to save at 10, if you make the changes to save ever 10 epochs OneTrainer will save when Epoch 9 is complete.
- Backups can be very large. Utilise rolling backups.
- Saves for fine-tunes (especially Stable Cascade or Pixart Alpha) require a lot of RAM (24 GB or more). This can cause the swap file to be used which will cause a decrease in performance and can cause saves to be very slow.