Hydra — a Multi-Task Learning Framework

Hydra is a flexible multi-task learning framework written in PyTorch 1.0. The following multi-objective optimization algorithms are implemented:

• Naive — a separate optimizer for each task
• Gradients averaging — average out the gradients to the network's body
• MGDA — described in the paper Multi-Task Learning as Multi-Objective Optimization (NIPS 2018)

A comprehensive survey on these algorithms (and more) can be found in this blog article.

Installation

• The code was written on Python 3.6. Clone this repository:

  git clone https://github.com/hav4ik/Hydra
  

• It is recommended to use anaconda for installation of core packages (since conda packages comes with low-level libraries that can optimize the runtime):

  conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
  conda install numpy pandas scikit-learn
  

• Some of the packages are not available from anaconda, so you can install them using pip:

  pip install -r requirements.txt
  

Getting started

• Examples of configuration files can be found here. A minimal example is available in starter.sh. Execute it as follows (will train with configurations in configs/toy_experiments/naive.yaml):

  ./starter.sh naive 50
  

Coming soon...

• Proper framework documentation and examples.