SAR3D: Autoregressive 3D Object Generation and Understanding via Multi-scale 3D VQVAE

TL;DR

SAR3D presents a scalable autoregressive framework for fast 3D object generation and rich 3D understanding by introducing a multi-scale 3D VQVAE that tokenizes 3D content into a latent tri-plane. By predicting the next scale rather than a single token, SAR3D achieves sub-second generation times and enables seamless integration with large language models through truncated-scale tokens for 3D captioning and understanding. The approach yields faster, higher-quality 3D generation compared with diffusion-based and other autoregressive methods, while enabling detailed multimodal descriptions via SAR3D-LLM. This work advances practical multimodal AI by unifying fast 3D generation with capable 3D understanding, paving the way for scalable 3D-aware applications in vision and language domains.

Abstract

Autoregressive models have demonstrated remarkable success across various fields, from large language models (LLMs) to large multimodal models (LMMs) and 2D content generation, moving closer to artificial general intelligence (AGI). Despite these advances, applying autoregressive approaches to 3D object generation and understanding remains largely unexplored. This paper introduces Scale AutoRegressive 3D (SAR3D), a novel framework that leverages a multi-scale 3D vector-quantized variational autoencoder (VQVAE) to tokenize 3D objects for efficient autoregressive generation and detailed understanding. By predicting the next scale in a multi-scale latent representation instead of the next single token, SAR3D reduces generation time significantly, achieving fast 3D object generation in just 0.82 seconds on an A6000 GPU. Additionally, given the tokens enriched with hierarchical 3D-aware information, we finetune a pretrained LLM on them, enabling multimodal comprehension of 3D content. Our experiments show that SAR3D surpasses current 3D generation methods in both speed and quality and allows LLMs to interpret and caption 3D models comprehensively.

Figures (12)

• Figure 1: Our method, SAR3D, proposes a comprehensive framework for 3D generation and understanding via autoregressive modeling. For (a) 3D generation, given a single image or text prompt, SAR3D generates multi-scale 3D objects in an autoregressive manner. For (b) 3D understanding, SAR3D-LLM can interpret a 3D model and provide a detailed description.
• Figure 2: Overview of Multi-scale VQVAE. Given a 3D model, we leverage multi-view RGB-D(epth) renderings and Plücker embeddings as the input to our multi-view encoder $\mathcal{E}$. The encoder predicts a continuous feature map that is then quantized by the multi-scale quantizer $\mathcal{Q}$, giving $R=(r_1, r_2, \dots, r_K)$ of latent tri-plane features. Each code of different scales share the same codebook. The triplane decoder then converts the quantized latent triplane features into the triplane representation through a plane-wise manner. The predicted triplane is multi-view supervised with the ground truth image, depth, and normal.
• Figure 3: Overview of 3D Generation and 3D Understanding. Given a 3D model, our 3D VQVAE encodes it into multi-scale discrete tokens for both 3D generation and understanding. In (a) 3D Generation, text or a single image is encoded by CLIP$_\text{T}$ or DINOv2, and the encoded condition features are integrated into the decoder-only transformer via cross attention. The transformer then causally predicts each scale of the latent triplane. In (b) 3D Understanding, truncated 3D tokens are first processed with an MLP projector. The large language model receives a multimodal sequence of text and 3D tokens and generates a detailed caption describing the input 3D model.
• Figure 4: Qualitative Comparison of Image-conditioned 3D Generation. Here, we compare with the state-of-the-art 3D generative models under different categories. As visualized here, our method achieves superior 3D consistency across views and generates intact objects without distortion. For more comparisons with other methods, please refer to the Supp Mat.
• Figure 5: More results of image and text conditioned 3D generation of SAR3D.