🔥 TAR3D: Creating High-Quality 3D Assets via Next-Part Prediction

[Paper]   [Project Page]   [Jittor Version]  [Demo]

🚩 Todo List

• Source code of 3D VQVAE.
• Source code of 3D GPT.
• Pretrained weights of 3D reconstruction.
• Pretrained weights of text-to-3D generation.
• Pretrained weights of image-to-3D generation.

⚙️ Setup

1. Dependencies and Installation

We recommend using Python>=3.10, PyTorch>=2.1.0, and CUDA>=12.1.

conda create --name tar3d python=3.10
conda activate tar3d
pip install -U pip

# Ensure Ninja is installed
conda install Ninja

# Install the correct version of CUDA
conda install cuda -c nvidia/label/cuda-12.1.0

# Install PyTorch and xformers
# You may need to install another xformers version if you use a different PyTorch version
pip install torch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 --index-url https://download.pytorch.org/whl/cu121
pip install xformers==0.0.22.post7

# For Linux users: Install Triton 
pip install triton

# Install other requirements
pip install -r requirements.txt

2. Downloading Datasets

3. Downloading Checkpoints

⚡ Quick Start

1. Reconstructing a 3D Geometry with 3D VQ-VAE

2. Text-to-3D Generation

3. Image-to-3D Generation

💻 Training

1. Training 3D VQ-VAE

2. Training Text-to-3D GPT

3. Training Image-to-3D GPT

💫 Evaluation

1. 2D Evaluation (PSNR, SSIM, Clip-Score, LPIPS)

2. 3D Evaluation (Chamfer Distance, F-Score)

🤗 Acknowledgements

We thank the authors of the following projects for their excellent contributions to 3D generative AI!

• Michelangelo
• InstantMesh
• OpenLRM
• 3DShape2VecSet

📚 BibTeX

If you find TAR3D useful for your research and applications, please cite using this BibTeX:

@article{zhang2024tar3d,
  title={TAR3D: Creating High-quality 3D Assets via Next-Part Prediction},
  author={Zhang, Xuying and Liu, Yutong and Li, Yangguang and Zhang, Renrui and Liu, Yufei and Wang, Kai, Ouyang, Wanli and Xiong, Zhiwei and Gao, Peng and Hou, Qibin and Cheng, Ming-Ming},
  journal={arXiv preprint arXiv:2412.16919},
  year={2024}
}