Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings

Aditya Sanghi, Aliasghar Khani, Pradyumna Reddy, Arianna Rampini, Derek Cheung, Kamal Rahimi Malekshan, Kanika Madan, Hooman Shayani

TL;DR

A novel approach called Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based, compact latent encodings, achieving an impressive 2427x compression ratio with minimal loss of detail and releasing the largest pretrained 3D generative models across different modalities.

Abstract

Large-scale 3D generative models require substantial computational resources yet often fall short in capturing fine details and complex geometries at high resolutions. We attribute this limitation to the inefficiency of current representations, which lack the compactness required to model the generative models effectively. To address this, we introduce a novel approach called Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based, compact latent encodings. Specifically, we compress a $256^3$ signed distance field into a $12^3 \times 4$ latent grid, achieving an impressive 2427x compression ratio with minimal loss of detail. This high level of compression allows our method to efficiently train large-scale generative networks without increasing the inference time. Our models, both conditional and unconditional, contain approximately one billion parameters and successfully generate high-quality 3D shapes at $256^3$ resolution. Moreover, WaLa offers rapid inference, producing shapes within two to four seconds depending on the condition, despite the model's scale. We demonstrate state-of-the-art performance across multiple datasets, with significant improvements in generation quality, diversity, and computational efficiency. We open-source our code and, to the best of our knowledge, release the largest pretrained 3D generative models across different modalities.

Wavelet Latent Diffusion (Wala): Billion-Parameter 3D Generative Model with Compact Wavelet Encodings

TL;DR

A novel approach called Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based, compact latent encodings, achieving an impressive 2427x compression ratio with minimal loss of detail and releasing the largest pretrained 3D generative models across different modalities.

Abstract

Large-scale 3D generative models require substantial computational resources yet often fall short in capturing fine details and complex geometries at high resolutions. We attribute this limitation to the inefficiency of current representations, which lack the compactness required to model the generative models effectively. To address this, we introduce a novel approach called Wavelet Latent Diffusion, or WaLa, that encodes 3D shapes into wavelet-based, compact latent encodings. Specifically, we compress a signed distance field into a latent grid, achieving an impressive 2427x compression ratio with minimal loss of detail. This high level of compression allows our method to efficiently train large-scale generative networks without increasing the inference time. Our models, both conditional and unconditional, contain approximately one billion parameters and successfully generate high-quality 3D shapes at resolution. Moreover, WaLa offers rapid inference, producing shapes within two to four seconds depending on the condition, despite the model's scale. We demonstrate state-of-the-art performance across multiple datasets, with significant improvements in generation quality, diversity, and computational efficiency. We open-source our code and, to the best of our knowledge, release the largest pretrained 3D generative models across different modalities.

Paper Structure

This paper contains 27 sections, 2 equations, 8 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Stage 1: Wavelet VQ-VAE
  5. Stage 2: Latent Diffusion model
  6. Inference
  7. Results
  8. Experimental Setup
  9. Experiments
  10. Point cloud-to-Mesh
  11. Voxel-to-Mesh
  12. Image-to-Mesh
  13. Conclusion
  14. Acknowledgement
  15. Additional Results and Details
  16. ...and 12 more sections

Figures (8)

  • Figure 2: WaLa generates 3D shapes across various input modalities (see appendix for more).
  • Figure 3: Overview of the WaLa network architecture and 2-stage training process and inference method. Top Left: Stage 1 autoencoder training, compressing diffusible wavelet tree ($W$) shape representation into a compact latent space. Top Right: Conditional/unconditional diffusion training. Bottom: Inference pipeline, illustrating sampling from the trained diffusion model and decoding the sampled latent into a Wavelet Tree ($W$), then into a mesh.
  • Figure 4: Qualitative comparison with other methods for single-view (top-left), multi-view (top-right), voxels (bottom-left), and point cloud (bottom-right) conditional input modalities. hui2024makeopenlrmTripoSR2024xu2024instantmeshtang2024lgmchen2024meshanythingpoint-e
  • Figure 5: The 6 different sketch types. From left to right: Grease Pencil, Canny, HED, HED+potrace, HED+scribble, CLIPaasso, and a depth map for reference. Mesh taken from fu20213d.
  • Figure 6: The 8 different views for which sketches were generated. Images created using the Grease Pencil technique on a mesh taken from fu20213d. The CLIPasso technique was only used on the first, fifth, and sixth views from the left.
  • ...and 3 more figures