How to speed up cache_latents with SDXL finetuning?

[a-l-e-x-d-s-9]

I have 33k images with SDXL finetuning, using A40 45GB Vram, cache_latents took me 2 hours with average 3.7 it/s.
I'm using vae_batch_size=0. When I tried 8/4, the it/s was even smaller.
Is there a way to speed up the process?

[araleza]

I'm no great expert, but looking through the section of code that caches latents in the train_util.py file, there's a bit that says this:

    # FIXME this slows down caching a lot, specify this as an option
    if torch.cuda.is_available():
        torch.cuda.empty_cache()

Maybe comment out those two lines and see if you get a speed boost? Those lines are probably there to help people who don't have much vram, which may not be useful for your 45GB card.