Can someone explain "repeats", "epochs" and total steps in Lora training?

[tommyjohn81]

What is the repeats variable? Why would I want to repeat the same 1mage more than once? This seems to inflate the number of steps required.

What is the formula for epochs based on repeats and total steps? I am accustomed to dreambooth training where I use 120* number of training images to get total steps. So if I have 10 images, I would train for 1200 steps. How would I get the equivalent using 10 images, repeats, steps and epochs for Lora?

If I want to get a checkpoint every 100 images, how would I set the "save n epochs ratio" based on the example above?

Thanks in advance!

[tommyjohn81]

Answering some of my own questions from this resource:

https://rentry.org/59xed3

Total number of steps = (# of images) x (# of repeats) x (number of epochs)

still not clear what the purpose of repeats is though.

[kohya-ss]

The steps calculation is a bit complex with bucketing.

The number of steps for an epoch for a certain bucket = ceil ((# of images in the bucket) x (# of repeats) / (batch size))
The number of steps for an epoch = sum of steps for all buckets
Total number of steps = (# of steps for an epoch) * (training epochs)

There are two purpose for repeats.

1. Balance the number of training images to the regularization images. The number of regularization images is larger than the training, so it is required to repeat training images for using all regularization images in the epoch.
2. Control 'weight' over folders. If you have high quality images and low quality images, you can set higher number of repeats for high quality images, and lower for low quality.

[DarksealStudios]

thank you, yes, I've seen the video and am subscribed, but have you compared the results of this verses the same steps without regularization? Did it make a visual difference?

[FurkanGozukara]

thank you, yes, I've seen the video and am subscribed, but have you compared the results of this verses the same steps without regularization? Did it make a visual difference?

I compared in this video : https://youtu.be/Tb4IYIYm4os

it wasnt kohya lora but logic is same

[DarksealStudios]

So, if then I am reading you right, comparing results from regularization vs non-regularization did NOT make a difference.

[FurkanGozukara]

So, if then I am reading you right, comparing results from regularization vs non-regularization did NOT make a difference.

It made. I have shown in video

[DarksealStudios]

thank you. I watched it when you made it weeks/months ago. I just skimmed though it again. I think if you were to try again with daDaptation you may find it no longer needed. I don't know why your images fried with so few steps and a low learning rate without reg images. I have not experienced the same issues with daD, but certainly did with 8bitAdam. I thought for a moment you had batch 12 making your 2400 really 29k, but then I saw the settings... I really think it's just the difference between kohya and dreambooth. Having used them both I have to pick kohya. I'm only on a 2070 with 8gb, but the results with the same dataset were better on kohya with no regularization images... not that I tried WITH.

[FurkanGozukara]

@kohya-ss can you change your repeating logic?

with your logic I am not able to use 40 different classification images per training image with having 1 step = 1 epoch

it is being 40 steps = 1 epoch

however with automatic1111 web ui dreambooth extension, i am able to set any number of classification images per instance and i am able to get checkpoint with even 1 step = 1 epoch

with your way this is happening

lets say I want to use 100 different class images for per training image. then the minimum checkpoint i can save is actually 100 epochs instead of 1 epoch

[kohya-ss]

Thank you for your contribution and for the great tutorials!

Unfortunately, it is currently quite difficult to change the repeat logic.
The reason for this is that there is no ability to split the data in a DataSet into multiple epochs. My understanding is that, depending on the configuration, the DataLoader will restart the process at each epoch, making it quite difficult to retain any information across epochs.
Aspect Ratio Bucketing also makes processing difficult. If there are class images of different resolutions, and if the DataSet is divided into epochs, the bucket configuration must be different for each epoch. This is very difficult to handle.

Therefore, the number of steps per epoch must be at least equal to the number of class images. I hope you understand this as a limitation for now.

[FurkanGozukara]

@kohya-ss thank you for reply. we are thankful to your repo and effort.