Batch size

Batch size increases speed, but requires more VRAM Higher batch size might need a higher learning rate

Grad Accumulation

Grad size 3, should be on paper, similar to batch 3 Grad 3 batch 1, will do 3 batches of size 1 but only apply the learning at the end of the 3 iteration. It will be the same speed as batch 1, but should have the training result of batch 3 So grad 3 batch 1 has an equivalent batch size of 3, training wise

Equivalent Batch Size

Grad 3 batch 2 => equivalent batch size 6 Gradient accumulation allows to replicate the results of high batch sizes(think of 48+ GB graphic card) on low VRAM environment. The trade off is speed. !!You want the equivalent batch size to be able to divide the training images and leave no remainder!! For example, for 77 images with no class images your only batch options are

Batch Size	Grad Size	Equivalent
1	1	1
1	7	7
1	11	11
1	77	77
7	1	7
7	11	77
11	1	11
11	7	77
77	1	77

Batch size suggestions

If speed is the main focus and VRAM is plenty, go for the highest batch size you are able to run(leaving no remainder).

High Batch Size

Training at high batch sizes (or equivalent) will produce a training that assimilates the features of the instance images more deeply. This is good for style but might results on weird generations. elephant at equivalent batch size 150(15*10) trained at 1.5e-4 (NO TENC)(150 captioned images, no class images used)

Low Batch Size

Low batch size (or equivalent) will produce images that usually maintain a higher integrity. This is good when training on pictures of yourself, or on a specific object.