About lora_vit.head bias

[zhongtao93]

Thank you for opensource code!
I have a question about lora_vit.head. It seems when save lora fc weight(

LoRA-ViT/lora.py

Line 100 in 45be249

fc_tensors = {f"fc_{_in}in_{_out}out": self.lora_vit.fc.weight}

) and switch lora fc weight(

LoRA-ViT/lora.py

Line 497 in 45be249

self.lora_vit.head.weight = Parameter(self.fc_loras[idx])

), only save and load linear weight but no bias. But when using lora_vit, it seems bias=True in melo.lora_vit.head (examples.ipynb)

, which cause a random init for head.bias. Is this a bug or what?
Hoping for your reply~

[Eyvaz27]

LoRA-ViT/lora.py

Line 244 in 45be249

w_a_linear_q = nn.Linear(self.dim, r, bias=False)

bias for added low rank layers are set to False, thus there is no need to save them in memory