xformers not working + [Feature Requests]

[Hergotzer]

When running the XL Lora-trainer, the following warning message is printed towards the end of printing dependencies:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
grpcio-status 1.62.3 requires protobuf>=4.21.6, but you have protobuf 3.20.3 which is incompatible.
peft 0.14.0 requires accelerate>=0.21.0, but you have accelerate 0.19.0 which is incompatible.
peft 0.14.0 requires huggingface-hub>=0.25.0, but you have huggingface-hub 0.22.0 which is incompatible.
sentence-transformers 3.3.1 requires transformers<5.0.0,>=4.41.0, but you have transformers 4.30.2 which is incompatible.

This does not appear to affect the training itself, however, as the program will still train normally. However, if trying to use xformers (which I have at least heard is better than SDPA, although this might be GPU-dependent and not necessarily as good on the free tier GPUs on Colab), the program fails to run, as it can't find a correct version of xformers to install.

Additionally, I have a couple of feature requests that I believe would be easy to implement and be quite beneficial:

• There is a newer version of Prodigy, which was released around December last year. It works just like the current one, but gives the users a couple of new handy tools to use, which drastically reduce VRAM usage: https://github.com/konstmish/prodigy

I have tested it myself on Kohya_ss, and it works completely fine. Recommend using slice_p=11 in the optimizer arguments for a nice VRAM boost, for practically no detriment to the Lora quality. Additionally, setting beta1=0 can further reduce VRAM usage, but this time with some quality loss. Simply updating the script to download this newer version should be enough.

• The other request I would have is Debiased Estimation loss, which has been implemented in Kohya_ss. It's basically a replacement for min_snr_gamma but requires no additional hyperparameters, making it somewhat simpler and more consistent to use, and while the effect is small, it also appears to have some beneficial effect on Lora quality.

Finally, I'd like to ask about the "multinoise"-setting. I gather it enables Multires noise, but what parameters are you using for it?

[uYouUs]

First of all, xformers for torch 2.5.1 is 0.0.29.post1

Secondly, the warning you see is not important. It basically warns that because pip downgraded packages to run Kohya, that those programs won't work properly. Which is totally fine because we aren't using them.

Third, adding the newer Prodigy might need an upgrade to other packages and possibly interactions with bitsandbytes. Might be possible, but probably a bit of work.

Fourth, if it is indeed in newer versions of Kohya, you might be able to use it with this link
https://colab.research.google.com/github/uYouUs/Hollowstrawberry-kohya-colab/blob/Experiments/Lora_Trainer_XL_V.ipynb
It's experimental but It is running the newest Kohya from the dev branch.

[Hergotzer]

Just did some testing with the new Notebook you linked;

1. The new Prodigy version works; Simply edit the code to install a newer version (1.1.2 at the moment) and add slice_p= to the optimizer arguments (recommended value for slice_p is 11, but other values can also work; lower values are more accurate but also don't give the same VRAM savings). However, the setting doesn't appear to save too much VRAM; For me, it only went from 14.5GB at batch 5 to 14.1GB. However, that's still enough to safely increase batch by 1, which is always nice. If you use Dim=16, you can even get batch 7. That is with the free tier.

2. While the new Notebook might have Debiased Estimation loss, I can't check if this is true and even if it was, I have no idea how to activate it in Colab.