This is the official PyTorch reimplementation of Polygon-RNN++ (CVPR 2018). This repository allows you to train new Polygon-RNN++ models, and run our demo tool on local machines. For technical details, please refer to:
Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++
David Acuna*, Huan Ling*, Amlan Kar*, Sanja Fidler (* denotes equal contribution)
CVPR 2018
[Paper] [Video] [Project Page] [Demo]
To get the code, please signup here. We will be using GitHub to keep track of issues with the code and to update on availability of newer versions (also available on website and through e-mail to signed up users).
If you use this code, please cite:
@inproceedings{AcunaCVPR18,
title={Efficient Interactive Annotation of Segmentation Datasets with Polygon-RNN++},
author={David Acuna and Huan Ling and Amlan Kar and Sanja Fidler},
booktitle={CVPR},
year={2018}
}
@inproceedings{CastrejonCVPR17,
title = {Annotating Object Instances with a Polygon-RNN},
author = {Lluis Castrejon and Kaustav Kundu and Raquel Urtasun and Sanja Fidler},
booktitle = {CVPR},
year = {2017}
}
These are the reproduction results from this repository as compared to the paper
| Training Type | Num first points | LSTM Beam Size | Before | Now |
|---|---|---|---|---|
| MLE + Att | 1 | 1 | 65.43 | 66.35 |
| MLE + Att + RL | 1 | 1 | 67.17 | 67.45 |
| MLE + Att + Evaluator | 5 | 1 | 69.72 | 71.05 |
| MLE + Att + Evaluator | 5 | 8 | 70.21 | 70.91 |
| MLE + Att + Evaluator + GGNN | 5 | 8 | 71.38 | 72.05 |
| MLE + Att + Evaluator + GGNN | 5 | 1 | - | 72.08 |
| MLE + Att + Evaluator + GGNN (Shared Encoder) | 5 | 8 | - | 72.22 |
| MLE + Att + Evaluator + GGNN (Shared Encoder) | 5 | 1 | - | 72.33 |
Note: Benchmarked forward pass speed for the tool (with 5 first points and 1 beam size) is 0.3 seconds per interaction on a TitanXp
Note: Shared Encoder refers to sharing the Resnet between the graph network and the convLSTM network. In the original paper, the two networks were kept separate.
All the code has been run and tested on Ubuntu 16.04, Python 2.7.12, Pytorch 0.4.0, CUDA 9.0, TITAN X/Xp and GTX 1080Ti GPUs
- Get code after signing up
- Go into the downloaded code directory
cd <path_to_downloaded_directory>
- Setup python environment
virtualenv env
source env/bin/activate
pip install -r requirements.txt
- Add the project to PYTHONPATH
export PYTHONPATH=$PWD
- Setup your environment
- Download the MLE+RL+Evaluator+GGNN model after getting access by signing up
- Launch backend (flask server) with,
python Tool/tool.py --exp Experiments/tool.json --reload <path_to_model> --port <port> --image_dir Tool/frontend/static/img/
- Edit Tool/frontend/static/js/polygon.js and change globalFolder to the appropriate directory based on where you cloned the repository.
- With python2.7, run
cd Tool/frontend/
python -m SimpleHTTPServer
- On your browser, navigate to localhost:8000. You should see a page like
Note: Replace SimpleHTTPServer with http.server if you are using python3 for the server
Note: You can setup your own image directory by editing Tool/frontend/static/js/polygon.js and passing that path to Tool/tool.py from the command line. This image directory MUST contain the pre-defined images that are defined in Tool/frontend/index.html
- Setup your environment
- Download pretrained models after getting access by signing up
python Scripts/prediction/generate_annotation.py --exp <path_to_corresponding_experiment> --reload <path_to_checkpoint> --output_dir <path_to_store_predictions>
- If you are testing a RL checkpoint, then the experiment file would correspond be the RL experiment file, and similarly for other stages of the model
- You can check predicted/GT masks for every instance in the output_dir
- To get scores, run
python Scripts/get_scores.py --pred <path_to_preds> --output <path_to_file_to_save_results>
- Download the Cityscapes dataset (leftImg8bit_trainvaltest.zip) from the official website [11 GB]
- Our processed annotation files are included in the download file you get after signing up
- From the root directory, run the following command with appropriate paths to get the annotation files ready for your machine
python Scripts/data/change_paths.py --city_dir <path_to_downloaded_leftImg8bit_folder> --json_dir <path_to_downloaded_annotation_file> --out_dir <output_dir>
To train on your custom datasets, you have one of two options:
- Prepare annotation data similar to our annotation files and use our default DataProvider
- Implement your own DataProvider following the cityscapes implementation for your own data
- Setup your environment
- Download the pre-trained Pytorch Resnet-50 from here
- Note - While resuming training, always resume from end of epoch checkpoints to produce reproducible results!
- Edit the experiment file at Experiments/mle.json and change paths for your machine
- From the root directory, run
python Scripts/train/train_ce.py --exp Experiments/mle.json --resume <optional_if_resuming_training>
- You can view progress on Tensorboard (logs are at <experiment_dir>/logs/)
- Edit the experiment file at Experiments/rl.json and change paths for your machine
- In the experiment file, set xe_initializer to the best MLE model
- From the root directory, run
python Scripts/train/train_rl.py --exp Experiments/mle.json --resume <optional_if_resuming_training>
- Note - You might have to play with hyperparameters a bit to achieve stable training, especially temperature, lr and lr_decay
- Edit the experiment file at Experiments/evaluator.json and change paths for your machine
- In the experiment file, set xe_initializer to the best RL model
- From the root directory, run
python Scripts/train/train_evaluator.py --exp Experiments/evaluator.json --resume <optional_if_resuming_training>
- Edit the experiment file at Experiments/ggnn.json and change paths for your machine
- In the experiment file, set xe_initializer to the best Evaluator model
- From the root directory, run
python Scripts/train/train_ggnn.py --exp Experiments/ggnn.json --resume <optional_if_resuming_training>