Patron

This repo contains the code for our ACL 2023 paper Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach.

Model Framework

Performance

The results on different datasets (using 128 labels as the budget) for fine-tuning are summarized as follows:

Method	IMDB	Yelp-full	AGNews	Yahoo!	DBPedia	TREC	Mean
Full Supervision (RoBERTa-base)	94.1	66.4	94.0	77.6	99.3	97.2	88.1
Random Sampling	86.6	47.7	84.5	60.2	95.0	85.6	76.7
Best Baseline (Chang et al. 2021)	88.5	46.4	85.6	61.3	96.5	87.7	77.6
Patron (Ours)	89.6	51.2	87.0	65.1	97.0	91.1	80.2

For prompt-based learning, we use the same pipeline as the LM-BFF. The result with 128 labels is shown as follows.

Method	IMDB	Yelp-full	AGNews	Yahoo!	DBPedia	TREC	Mean
Full Supervision (RoBERTa-base)	94.1	66.4	94.0	77.6	99.3	97.2	88.1
Random Sampling	87.7	51.3	84.9	64.7	96.0	85.0	78.2
Best Baseline (Yuan et al., 2020)	88.9	51.7	87.5	65.9	96.8	86.5	79.5
Patron (Ours)	89.3	55.6	87.8	67.6	97.4	88.9	81.1

Dependencies

python 3.8
transformers==4.2.0
pytorch==1.8.0
scikit-learn
faiss-cpu==1.6.4
sentencepiece==0.1.96
tqdm>=4.62.2
tensorboardX
nltk
openprompt

Datasets

We use the following four datasets for the main experiments.

Dataset	Task	Number of Classes	Number of Unlabeled Data/Test Data
IMDB	Sentiment	2	25k/25k
Yelp-full	Sentiment	5	39k/10k
AG News	News Topic	4	119k/7.6k
Yahoo! Answers	QA Topic	5	180k/30.1k
DBPedia	Ontology Topic	14	280k/70k
TREC	Question Topic	6	5k/0.5k

The processed data can be found at this link. The folder to put these datasets will be discribed in the following parts.

Data Selection Pipeline

a) Generating Unsupervised Text Embeddings via SimCSE

Run the following commands

python gen_embedding_simcse.py --dataset [the dataset you use] --gpuid [the id of gpu you use] --batchsize [the number of data processed in one time]

b) Estimating Uncertainty via Prompts

We provide the pseudo prediction obtained via prompts in the above link for datasets. Please refer to the original papers for details.

c) Data Selection with Uncertainty Propagation and PTR

Run the following commands (example on AG News dataset)

python patron_sample.py --dataset agnews --k 50 --rho 0.01 --gamma 0.5 --beta 0.5

Some important hyperparameters:

• rho: the parameter used for uncertainty propagation in Eq. 6 of the paper
• beta: the regularization of distance in Eq. 8 of the paper
• gamma: the weight of the regularization term in Eq. 10 of the paper

Experiments

Running Fine-tuning Experiments

See finetune folder for detailed instructions.

Running Prompt-based Learning Experiments

See prompt_learning folder for detailed instructions.

Running on a New Dataset

See this link as the pipeline for generating the prompt-based predictions. Note that you need to customize your prompt verbalizers and templates.

To generate the document embeddings, you can follow the above commands by using SimCSE.

Once you generate the index for the selected data, then you can use the pipelines in Running Fine-tuning Experiments and Running Prompt-based Learning Experiments for the few-shot fine-tuning and prompt-based learning experiments.

Citation

Please kindly cite the following paper if you find this repo helpful for your research. Thanks in advance!

@article{yu2022patron,
  title={Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approach
},
  author={Yue Yu and Rongzhi Zhang and Ran Xu and Jieyu Zhang and Jiaming Shen and Chao Zhang},
  journal={arXiv preprint arXiv:2209.06995},
  year={2022}
}

Acknowledgements

We would like to thank the authors from the repo SimCSE and OpenPrompt for the well-organized code.