Backup and Save - Nerogar/OneTrainer GitHub Wiki
The Backup Tab
As models evolve larger and more complex and larger consumer GPUs become available, it is important to clarify between a backup and a save.
- Backup - This is a backup of all of the internals that OneTrainer utilizes at a period of time. It is intended to be something that you can utilize to restart OneTrainer at this point. Changing certain functions, may require input from Nero or some others to get the exact changes on how to do this. The primary use case for backups is FineTunes with hours and hours of work where you do not want to start from the beginning again if things go different than you want.
- Save - If you want to test your work during intermediate training steps, then these are the settings you want to focus on.
- Backup and save are located in the workspace directory.
Backup Focused Settings
- Backup after X Y. This will create a backup after X Epochs/Steps/Second/Minute/Hour/Never/Always. Note that Never and Always are special and will ignore your integer.
- Backup now. Pressing this button will tell OneTrainer to do a backup at the next opportunity.
- Rolling Backup - A storage saving setting. Enabling this will automatically delete backups as new ones are created.
- Rolling Backup Count - The number of backups OneTrainer will keep. Older ones will automatically be deleted if Rolling Backup is enabled.
- Backup Before Save - This setting ties backup creation to save creation. This can used in conjunction with your other backup settings, or you can use your save settings to control the backup creation.
Save Focused Settings
- Save every X Y. This will create a save every X Epochs/Steps/Second/Minute/Hour/Never/Always. Note that Never and Always are special and will ignore your integer.
- Skip First X: Skip the first X intervals (Epochs/Steps/Second/Minute/Hour). Ex: save every 10 epochs and skip first 100 will start saving after 110 epochs.
- Save Filename Prefix. A requested feature to help better sort saves. This will add a prefix to the standard OneTrainer save format.
Notes and Comments
- Changes to this tab will be reflected in a running training. For example if you are on epoch 9, and want to save at 10, if you make the changes to save ever 10 epochs OneTrainer will save when Epoch 9 is complete.
- Backups can be very large and can take up a lot of space very quickly.
- Saves for FineTunes (especially Stable Cascade or Pixart Alpha) require a lot of RAM (24 GB or more). This can cause the swap file to be used which will cause a decrease in performance and can cause saves to be very slow.