Add QLoRA

[anwai98]

This PR adds finetuning SAM using QLoRA.

TLDR: The in-place operation in LoRA's forward pass was creating the fuss for us. It is taken care of now ;)

PS. QLoRA works now and the loss and metric looks as desired (checked it for a couple of epochs now). There still are two major concerns: 1) QLoRA currently only works in full precision training, 2) the memory engagement is the same as LoRA :/

cc: @caroteu @constantinpape

[anwai98]

Okay, QLoRA works as expected now (now with mixed precision as well)! 🥳

PS. The bitsandbytes optimizers cannot handle the precision on which AMP allows gradient calculations. This created some of the missing fuss. And the PyTorch optimizers apparently can handle this easily.

PPS. The memory advantage over LoRA is super minor. I do see some advantage when trying it over vit_h but it's not the case for smaller ViTs.

I'll run a full-scale training on LIVECell and perform a final validation to confirm stuff. The metric and loss looks good so far!

[anwai98]

Here are the memory engagement mentions while training.