Unusual Model Utility Gap: Gradient Difference vs Ascent (Llama2)

[jeongjin0]

Hi, I noticed that for Llama2(forget10), gradient difference shows much lower model utility (~0.27) than gradient ascent (~0.63) in the leaderboard. This seems unusual since gradient difference is designed to maintain performance on the retain set while unlearning.

Interestingly, in Phi model results, gradient difference shows higher utility than ascent as expected. Could you help explain this significant performance gap in Llama2 implementation?

Thanks!

[zhilif]

Apologize for the confusing. Previously there's a bug in the leaderboard csv generation. We have uploaded a new version. Meanwhile, I will rerun evals again to double check. Thanks for your comments!