AutoSDF: Shape Priors for 3D Completion, Reconstruction and Generation

TL;DR

AutoSDF introduces a non-sequential autoregressive prior over a discretized 3D shape latent space learned with a patch-wise VQ-VAE, enabling flexible conditioning on arbitrary partial observations. A transformer models the latent distribution p(Z) with random orders, allowing shape completion, multi-view reconstruction, and language-guided generation to share a single prior across tasks. The prior is combined with task-specific naive conditionals to form conditional distributions, enabling diverse, high-quality outputs with limited paired data. Across ShapeNet and Pix3D, AutoSDF achieves competitive or superior results to task-specific baselines in shape completion, single-view prediction, and language-conditioned generation, demonstrating the value of a unified, learnable 3D shape prior for multimodal 3D inference.

Abstract

Powerful priors allow us to perform inference with insufficient information. In this paper, we propose an autoregressive prior for 3D shapes to solve multimodal 3D tasks such as shape completion, reconstruction, and generation. We model the distribution over 3D shapes as a non-sequential autoregressive distribution over a discretized, low-dimensional, symbolic grid-like latent representation of 3D shapes. This enables us to represent distributions over 3D shapes conditioned on information from an arbitrary set of spatially anchored query locations and thus perform shape completion in such arbitrary settings (e.g., generating a complete chair given only a view of the back leg). We also show that the learned autoregressive prior can be leveraged for conditional tasks such as single-view reconstruction and language-based generation. This is achieved by learning task-specific naive conditionals which can be approximated by light-weight models trained on minimal paired data. We validate the effectiveness of the proposed method using both quantitative and qualitative evaluation and show that the proposed method outperforms the specialized state-of-the-art methods trained for individual tasks. The project page with code and video visualizations can be found at https://yccyenchicheng.github.io/AutoSDF/.

Figures (16)

• Figure 1: Our approach combines a novel, non-sequential autoregressive prior, capturing the distribution over 3D shapes, with task-specific conditionals, to generate multiple plausible and high-quality shapes consistent with input conditioning. We show the efficacy of our approach across diverse tasks such as shape completion, single-view reconstruction and language-guided generation.
• Figure 2: Overview of Autoregressive Modeling.(top) We use a VQ-VAE to extract a low-dimensional discrete representation of 3D shapes. Using a patch-wise encoder enables independently encoding local context and allows downstream tasks with partial observations. (bottom) We learn a transformer-based autoregressive model over the latent representation. Using randomized sampling orders allows learning a 'non-sequential' autoregressive shape prior that can condition on arbitrary sets of partial latent observations.
• Figure 3: Overview of conditional generation. The proposed autoregressive prior can be used across diverse conditional generation tasks. For each task, we use a domain specific encoder followed by 3D up-convolutions to learn task specific conditional distributions. During inference, we can sample from the product distribution of the predicted conditionals and the learned autoregressive prior.
• Figure 4: Comparative results for Shape Completion. Given the partial inputs, we visualize the generated results from different methods. Our approach yields more diverse generations, while also better preserving the originally observed structure. For example, in the first row, given 4 slanted legs of a chair, some MPC generations make them straighter in the full point cloud, while they are preserved in our approach
• Figure 5: Qualitative results for Shape Completion. Our proposed approach is able to generate diverse plausible 3D shapes consistent with the partial input. The generated shapes are visually consistent with realistic shapes even with significantly missing parts( in Red)