3D-WAG: Hierarchical Wavelet-Guided Autoregressive Generation for High-Fidelity 3D Shapes

TL;DR

3D-WAG reframes autoregressive 3D shape generation as next-scale token map prediction over compact wavelet representations, enabling high-fidelity generation with lower inference cost than token-by-token AR or diffusion methods. The framework uses a two-stage training pipeline: a 3D VQ-VAE learns multi-scale wavelet token maps, then a GPT-2–style transformer autoregressively predicts higher-resolution token maps conditioned on previous scales, enabling coarse-to-fine geometry reconstruction via wavelet inversion. Key contributions include the compact wavelet representation, a multi-scale tokenization and reconstruction scheme with a detail-focused loss, and a next-scale AR objective that yields superior ShapeNet and DeepFashion3D results with faster generation. The approach demonstrates strong unconditional and conditional generation performance, offering a scalable, controllable path for high-fidelity 3D shape synthesis with practical inference speeds.

Abstract

Autoregressive (AR) models have achieved remarkable success in natural language and image generation, but their application to 3D shape modeling remains largely unexplored. Unlike diffusion models, AR models enable more efficient and controllable generation with faster inference times, making them especially suitable for data-intensive domains. Traditional 3D generative models using AR approaches often rely on ``next-token" predictions at the voxel or point level. While effective for certain applications, these methods can be restrictive and computationally expensive when dealing with large-scale 3D data. To tackle these challenges, we introduce 3D-WAG, an AR model for 3D implicit distance fields that can perform unconditional shape generation, class-conditioned and also text-conditioned shape generation. Our key idea is to encode shapes as multi-scale wavelet token maps and use a Transformer to predict the ``next higher-resolution token map" in an autoregressive manner. By redefining 3D AR generation task as ``next-scale" prediction, we reduce the computational cost of generation compared to traditional ``next-token" prediction models, while preserving essential geometric details of 3D shapes in a more structured and hierarchical manner. We evaluate 3D-WAG to showcase its benefit by quantitative and qualitative comparisons with state-of-the-art methods on widely used benchmarks. Our results show 3D-WAG achieves superior performance in key metrics like Coverage and MMD, generating high-fidelity 3D shapes that closely match the real data distribution.

Figures (9)

• Figure 1: 3D-WAG overview. We propose an autoregressive approach to 3D shape generation, based on the "next-scale" prediction paradigm. Left: Our hierarchical, multi-scale wavelet encoding enables the efficient generation of highly detailed surfaces, outperforming prior models (e.g, UDiFF zhou2024udiff, in the red box) for comparable latent code dimensionality, and showcasing generalization in conditional 3D generation tasks, like text-to-3D. Right: Unconditional generation of diverse, high-fidelity shapes, supporting varied implicit spatial representations using our approach.
• Figure 2: Overview of our architecture. We follow a two-stage training approach that is standard with latent generative models. In Stage 1, we train a vector-quantized autoencoder (VQ-VAE) on the wavelet volumes with multi-scale patch-wise quantization tian2024visual in the latent space giving us quantized feature maps $\hat{z}_1, \hat{z}_2, \ldots, \hat{z}_K$. In Stage 2, the multi-scale VQ-VAE codebook indices in the form of token maps $f_1, f_2, \ldots, f_K$ are flattened and learned with an autoregressive decoder-only Transformer prior model, enabling next-scale generation. During inference, the generated token maps are reshaped and mapped to obtain multi-scale quantized features by using codebook. Then the obtained features are converted into wavelet volumes by the decoder from Stage 1. The wavelet volumes can be further converted into implicit functions using wavelet inversion.
• Figure 3: 3D chair generation results on ShapeNet, comparing IM-GAN, WaveGen, SPAGHETTI, UDiff, and SALAD, with our 3D-WAG on the far right. 3D-WAG shows improved structural coherence, finer details, and a wider diversity of generated designs, capturing both simple and complex geometries more effectively than other methods.
• Figure 4: 3D garment generation results on the DeepFashion3D dataset zhu2020deep, comparing PointDiff, WaveGen, LAS-Diffusion, UDiff, and our method (3D-WAG) on the far right. 3D-WAG produces more coherent and realistic garment shapes, capturing intricate structural details such as folds, sleeves, and openings with higher fidelity than other methods.
• Figure 5: Examples of unconditional generation examples under the ShapeNet chang2015shapenet dataset.