Skip to content

Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)

Notifications You must be signed in to change notification settings

xtk8532704/UVTR

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 

Repository files navigation

UVTR

arXiv visitors

Unifying Voxel-based Representation with Transformer for 3D Object Detection

Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, Jiaya Jia

[arXiv] [BibTeX]


This project provides an implementation for the NeurIPS 2022 paper "Unifying Voxel-based Representation with Transformer for 3D Object Detection" based on mmDetection3D. UVTR aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection.

Preparation

This project is based on mmDetection3D, which can be constructed as follows.

cp -r projects mmdetection3d/
cp -r extra_tools mmdetection3d/
  • Prepare the nuScenes dataset following the structure.
  • Generate the unified data info and sampling database for nuScenes dataset:
python3 extra_tools/create_data.py nuscenes --root-path ./data/nuscenes --out-dir ./data/nuscenes --extra-tag nuscenes_unified

Training

You can train the model following the instructions. You can find the pretrained models here if you want to train the model from scratch. For example, to launch UVTR training on multi GPUs, one should execute:

cd /path/to/mmdetection3d
bash extra_tools/dist_train.sh ${CFG_FILE} ${NUM_GPUS}

or train with a single GPU:

python3 extra_tools/train.py ${CFG_FILE}

Evaluation

You can evaluate the model following the instructions. For example, to launch UVTR evaluation with a pretrained checkpoint on multi GPUs, one should execute:

bash extra_tools/dist_test.sh ${CFG_FILE} ${CKPT} ${NUM_GPUS} --eval=bbox

or evaluate with a single GPU:

python3 extra_tools/test.py ${CFG_FILE} ${CKPT} --eval=bbox

nuScenes 3D Object Detection Results

We provide results on nuScenes val set with pretrained models.

NDS(%) mAP(%) mATE↓ mASE↓ mAOE↓ mAVE↓ mAAE↓ download
Camera-based
UVTR-C-R50-H5 40.1 31.3 0.810 0.281 0.486 0.793 0.187 GoogleDrive
UVTR-C-R50-H11 41.8 33.3 0.795 0.276 0.452 0.761 0.196 GoogleDrive
UVTR-C-R101 44.1 36.1 0.761 0.271 0.409 0.756 0.203 GoogleDrive
UVTR-CS-R50 47.2 36.2 0.756 0.276 0.399 0.467 0.189 GoogleDrive
UVTR-CS-R101 48.3 37.9 0.739 0.267 0.350 0.510 0.200 GoogleDrive
UVTR-L2C-R101 45.0 37.2 0.735 0.269 0.397 0.761 0.193 GoogleDrive
UVTR-L2CS3-R101 48.8 39.2 0.720 0.268 0.354 0.534 0.206 GoogleDrive
LiDAR-based
UVTR-L-V0075 67.6 60.8 0.335 0.257 0.303 0.206 0.183 GoogleDrive
Multi-modality
UVTR-M-V0075-R101 70.2 65.4 0.333 0.258 0.270 0.216 0.176 GoogleDrive

Acknowledgement

We would like to thank the authors of mmDetection3D and DETR3D for their open-source release.

License

UVTR is released under the Apache 2.0 license.

Citing UVTR

Consider cite UVTR in your publications if it helps your research.

@inproceedings{li2022uvtr,
  title={Unifying Voxel-based Representation with Transformer for 3D Object Detection},
  author={Li, Yanwei and Chen, Yilun and Qi, Xiaojuan and Li, Zeming and Sun, Jian and Jia, Jiaya},
  booktitle={Advances in Neural Information Processing Systems},
  year={2022}
}

About

Unifying Voxel-based Representation with Transformer for 3D Object Detection (NeurIPS 2022)

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 99.7%
  • Shell 0.3%