Why not reproject the internal Adam states during update_proj_gap?

[liuliu]

Hi, great project. After reading the paper and the implementation, I am wondering if it is considered to reproject the Adam internal states (exp_avg, exp_avg_sq) from previous subspace to the new subspace?

[liuliu]

The reproj momentum is mentioned in the FLoRA paper.

[jiaweizzhao]

Hi, thanks for the suggestion. We didn't include reprojection in the paper but will try to implement it in the repo.