MTTL - Multi-Task Transfer Learning
MTTL supports Python 3.8
and Python 3.9
. It is recommended to create a virtual environment for MTTL using virtualenv
or conda
. For example, with conda
:
conda create -n mttl python=3.9
conda activate mttl
Install the required Python packages:
pip install -e .
Please ensure that you have navigated to the projects/mhr
directory before running the Multi-Head Adapter Routing scripts:
cd projects/mhr
Download and prepare the datasets for the experiments using the following script:
bash datasets/create_datasets.sh
Based on your experiments, you may need to export one or more of the following environment variables:
T0_DATA_DIR: `data/t0_data/processed` if you ran the `create_datasets.sh`
NI_DATA_DIR: `data/ni_data/processed` if you ran the `create_datasets.sh`
XFIT_DATA_DIR: `data/ni_data/processed` if you ran the `create_datasets.sh`
CHECKPOINT_DIR
OUTPUT_DIR
CACHE_DIR
STORYCLOZE_DIR: path to your downloaded `.csv` files. See [the storycloze official website](https://cs.rochester.edu/nlp/rocstories/)
The general command for pre-training a model is:
python pl_train.py -c $CONFIG_FILES -k $KWARGS
Multiple CONFIG_FILES
can be concatenated as file1+file2
. To modify defaults, KWARGS
can be expressed as key=value
.
You can check scripts/pretrain for examples.
To perform finetuning for a test task, use the script pl_finetune.py
To perform an hyperparameter search for a test task, use the script pl_finetune_tune.py
.
The script will just call the functions in pl_finetune.py
in a loop. The script itself defines hp ranges for different fine-tuning types.
Alternatively, you can run the pre-configured scripts from the scripts
folder. For example:
bash scripts/mhr_pretrain.sh
If you run into issues with protoc TypeError: Descriptors cannot not be created directly.
, you can try to downgrade protobuf to 3.20.*:
pip install protobuf==3.20.*
pip install -e ".[test]"
pytest -vv tests
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.