Skip to content

google/lassie

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

This is not an officially supported Google product.

LASSIE: Learning Articulated Shapes from Sparse Image Ensemble via 3D Part Discovery (NeurIPS 2022)

Implementation for LASSIE. A novel method which estimates camera pose, 3D articulation, and part shapes of animal bodies given sparse images in-the-wild.

Chun-Han Yao2, Wei-Chih Hung2, Yuanzhen Li2, Michael Rubinstein2, Ming-Hsuan Yang2
, Varun Jampani2
1UC Merced, 2Waymo, 2Google Research, 2Yonsei University

Setup

A python virtual environment is used for dependency management. The code is tested with Python 3.7, PyTorch 1.11.0, CUDA 11.3. First, to install PyTorch in the virtual environment, run:

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 --extra-index-url https://download.pytorch.org/whl/cu113

Then, install other required packages by running:

pip install -r requirements.txt

Preparing data and pre-trained model

Pascal-part

  • Download Pascal images here and place them in data/pascal_part/JPEGImages/.
  • Download Pascal-part annotations here and place them in data/pascal_part/Annotations_Part/.
  • Download Pascal-part image sets here and place them in data/pascal_part/image-sets/.

Our image ensembles (web images)

  • Download images here and place them in data/web_images/images/.
  • Download keypoint annotations here and place them in data/web_images/annotations/.

Pre-trained primitive part decoder

  • Download pre-trained model here and place it in model_dump/.

LASSIE optimization

To run LASSIE optimization on sparse images of an animal class (e.g. zebra), simply run:

python train.py --cls zebra

The supported animal classes include: zebra, giraffe, tiger, elephant, kangaroo, penguin, horse, cow, sheep. The qualitative results can be found in results/zebra/. The optimization settings and initial 3D skeleton can be changed in main/config.py and main/skeleton.py, respectively. Note that the first time running LASSIE optimization could take a few minutes in the DINO feature clustering step.

Evaluation

Once optimization is completed, quantitative evaluation can be done by running:

python eval.py --cls zebra

The results will be stored in results/eval/zebra.txt.

Citation

@inproceedings{yao2022-lassie,
  title         = {{LASSIE}: {L}earning {A}rticulated {S}hape from {S}parse {I}mage {E}nsemble via 3D Part Discovery},
  author        = {Yao, Chun-Han and Hung, Wei-Chih and Li, Yuanzhen and Rubinstein, Michael and Yang, Ming-Hsuan and Jampani, Varun},
  booktitle     = {Advances in Neural Information Processing Systems (NeurIPS)},
  year          = {2022},
}

About

No description, website, or topics provided.

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages