VAR-3D: View-aware Auto-Regressive Model for Text-to-3D Generation via a 3D Tokenizer

TL;DR

VAR-3D tackles the challenging problem of text-to-3D generation by introducing a view-aware 3D VQ-VAE that produces discrete multi-scale triplane tokens, paired with a view-aware autoregressive generator trained under rendering-based supervision. This coupling mitigates information loss from quantization and aligns token predictions with visual fidelity and structural coherence, improving text–shape alignment and geometric accuracy. Empirical results on a large Objaverse-derived dataset show superior reconstruction quality (PSNR/SSIM) and distribution-level metrics (FID/KID) as well as competitive text-conditioned alignment (CLIP-T) compared with prior methods. The work demonstrates the practical impact of integrating cross-view-aware discretization with rendering-guided training for scalable, high-fidelity text-to-3D synthesis, while highlighting avenues for end-to-end and multimodal extensions.

Abstract

Recent advances in auto-regressive transformers have achieved remarkable success in generative modeling. However, text-to-3D generation remains challenging, primarily due to bottlenecks in learning discrete 3D representations. Specifically, existing approaches often suffer from information loss during encoding, causing representational distortion before the quantization process. This effect is further amplified by vector quantization, ultimately degrading the geometric coherence of text-conditioned 3D shapes. Moreover, the conventional two-stage training paradigm induces an objective mismatch between reconstruction and text-conditioned auto-regressive generation. To address these issues, we propose View-aware Auto-Regressive 3D (VAR-3D), which intergrates a view-aware 3D Vector Quantized-Variational AutoEncoder (VQ-VAE) to convert the complex geometric structure of 3D models into discrete tokens. Additionally, we introduce a rendering-supervised training strategy that couples discrete token prediction with visual reconstruction, encouraging the generative process to better preserve visual fidelity and structural consistency relative to the input text. Experiments demonstrate that VAR-3D significantly outperforms existing methods in both generation quality and text-3D alignment.

Figures (4)

• Figure 1: Overall architecture of the proposed VAR-3D framework. (a) 3D VQ-VAE: Multi-view renderings obtained from the 3D asset are encoded with dual self-attention to enhance feature interaction across views. Subsequently, the multi-view features are fused with multi-scale features from the downsampling stage and quantized to obtain latent triplane features $F=(f_1,\dots,f_r)$, which are decoded to reconstruct the 3D asset by a triplane decoder. (b) VAR model: In text-to-3D generation, the text is encoded into text features by CLIP-T. These text features are used as prefilling tokens for text conditional generation. Then the token sequence is progressively predicted across scales, and triplane features are synthesized via a codebook lookup to generate the 3D asset. Besides, a frozen triplane decoder provides visual supervision during training.
• Figure 2: Visual comparison of 3D reconstruction. We present reconstruction 3D objects generated by our 3D VQ-VAE displaying two views of each sample. Compared to the baseline methods, our approach consistently yields better quality regarding geometry, texture. Furthermore, our method also demonstrates better performance in terms of multi-view consistency.
• Figure 3: Qualitative comparison of text-conditioned 3D generation. We present text-to-3D generation results for objects generated by our method, displaying two views of each sample. Compared to the baseline methods, our approach consistently yields better quality regarding geometry, texture, and text–3D alignment.
• Figure 4: Visual comparison of our 3D VQ-VAE ablation experiments on the effectiveness of our components