[8 Sept 2024] I am aware the data download links are broken. the fix for this is being tracked in this issue.

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging

This repository contains code to analyze and predict contact maps for human grasping, presented in the paper

ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging - Samarth Brahmbhatt, Cusuh Ham, Charles C. Kemp, and James Hays, CVPR 2019

Paper (CVPR 2019 Oral) | Explore the dataset | Poster | Slides

Please see contactdb_utils for access to raw ContactDB data, and code to process it; analysis branch for code to generate various analysis graphs from the paper.

Setup

1. Download and install Miniconda (Python 3.x version).
2. Download this repository: git clone https://github.com/samarth-robo/contactdb_prediction.git. Commands for the following steps should be executed from the contactdb_prediction directory.
3. Create the contactdb_prediction environment: conda create env -f environment.yml, and activate it: source activate contactdb_prediction.
4. Download the preprocessed contact maps from this Dropbox link (17.9 GB). If the download location is CONTACTDB_DATA_DIR, make a symlink to it: ln -s CONTACTDB_DATA_DIR data/voxelized_meshes.
5. Download the trained models from this Dropbox link (700 MB). If the download location is CONTACTDB_MODELS_DIR, make a symlink to it: ln -s CONTACTDB_MODELS_DIR data/checkpoints.
6. (Optional, for comparison purposes): Download the predicted contact maps from this Dropbox link.

Predicting Contact Maps

We propose two methods to make diverse contact map predictions: DiverseNet and Stochastic Multiple Choice Learning (sMCL). This branch has code for the diversenet models. Checkout the smcl branch for sMCL code.

Predict contact maps for the 'use' instruction, using the voxel grid 3D representation:

$ python eval.py --instruction use --config configs/voxnet.ini --checkpoint data/checkpoints/use_voxnet_diversenet_release/checkpoint_model_86_val_loss\=0.01107167.pth
pan error = 0.0512
mug error = 0.0706
wine_glass error = 0.1398

You can add the --show object <pan | mug | wine_glass> flag to show the 10 diverse predictions:

$ python eval.py --instruction use --config configs/voxnet.ini --checkpoint data/checkpoints/use_voxnet_diversenet_release/checkpoint_model_86_val_loss\=0.01107167.pth --show_object mug
mug error = 0.0706

In general, the command is

python eval.py --instruction <use | handoff> --config <configs/voxnet.ini | configs/pointnet.ini> --checkpoint <checkpoint filename>

Use the following checkpoints:

Method	Checkpoint
Use - VoxNet	data/checkpoints/use_voxnet_diversenet_release/checkpoint_model_86_val_loss=0.01107167.pth
Use - PointNet	data/checkpoints/use_pointnet_diversenet_release/checkpoint_model_29_val_loss=0.6979221.pth
Handoff - VoxNet	data/checkpoints/handoff_voxnet_diversenet_release/checkpoint_model_167_val_loss=0.01268427.pth
Handoff - PointNet	data/checkpoints/handoff_pointnet_diversenet_release/checkpoint_model_745_val_loss=0.5969936.pth

Training your own models

Start the visdom server

$ source activate contactdb_prediction
$ visdom

The base training command is

python train_val.py --instruction <use | handoff> --config <configs/voxnet.ini | configs/pointnet.ini> [--device <GPU ID> --checkpoint_dir <directory where checkpints are saved> --data_dir <directory where data is downloaded>]

Citation

@inproceedings{brahmbhatt2018contactdb,
  title={{ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging}},
  author={Samarth Brahmbhatt and Cusuh Ham and Charles C. Kemp and James Hays},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2019},
  note={\url{https://contactdb.cc.gatech.edu}}
}