On the Effectiveness of Parameter-Efficient Fine-Tuning

Zihao Fu,¹ Haoran Yang,² Anthony Man-Cho So,² Wai Lam,² Lidong Bing,³ Nigel Collier¹

¹Language Technology Lab, University of Cambridge
²The Chinese University of Hong Kong
³DAMO Academy, Alibaba Group

[Paper (Full+Appendix)] [Slides] [Video]

Takeaways

• This paper gives a comprehensive explanation of why parameter-efficient models (such as Adapters, LoRA, Bitfit, etc.) achieve promising results.
• This paper unveils how the sparsity itself improves the model stability and generalization capability theoretically and empirically.
• This paper proposes a provable approximately best method to choose the tunable parameters for parameter-efficient models.

Install

This code is the SAM model proposed in the paper. We suggest to create a new conda env to install the dependencies.

git clone https://github.com/fuzihaofzh/AnalyzeParameterEfficientFinetune.git
cd AnalyzeParameterEfficientFinetune 
./scripts/install.sh

Run

Run the following code to train our SAM model on the CoLA dataset.

./scripts/train.sh