L3DG: Latent 3D Gaussian Diffusion

TL;DR

L3DG, the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation, is proposed, which significantly improves visual quality over prior work on unconditional object-level radiance field synthesis and demonstrates its applicability to room-scale scene generation.

Abstract

We propose L3DG, the first approach for generative 3D modeling of 3D Gaussians through a latent 3D Gaussian diffusion formulation. This enables effective generative 3D modeling, scaling to generation of entire room-scale scenes which can be very efficiently rendered. To enable effective synthesis of 3D Gaussians, we propose a latent diffusion formulation, operating in a compressed latent space of 3D Gaussians. This compressed latent space is learned by a vector-quantized variational autoencoder (VQ-VAE), for which we employ a sparse convolutional architecture to efficiently operate on room-scale scenes. This way, the complexity of the costly generation process via diffusion is substantially reduced, allowing higher detail on object-level generation, as well as scalability to large scenes. By leveraging the 3D Gaussian representation, the generated scenes can be rendered from arbitrary viewpoints in real-time. We demonstrate that our approach significantly improves visual quality over prior work on unconditional object-level radiance field synthesis and showcase its applicability to room-scale scene generation.

Figures (7)

• Figure 1: L3DG method overview: our 3D Gausssian compression model learns to compress 3D Gaussians into sparse quantized features using sparse convolutions and vector-quantization at the bottleneck (VQ-VAE). This allows our 3D diffusion model to efficiently operate on the compressed latent space. At test time, novel scenes are generated by denoising in latent space, which can be sparsified and decoded to high quality 3D Gaussians.
• Figure 2: Comparison on PhotoShape photoshape2018. Our method generates more detail than the baselines, such as thin structures, and has fewer artifacts.
• Figure 3: Comparison on ABO collins2022abo. Tables generated by our method are sharper and show less artifacts compared to the baselines.
• Figure 4: Ablation study on ABO collins2022abo. Training the 3D Gaussian compression model without the rendering losses (perceptual or RGB) leads to more blurry results, especially without perceptual loss. The variant without RGB loss additionally produces less color variations in the generated scenes.
• Figure 5: Qualitative results on unconditional room generation on 3D-FRONT fu20213dfront. Our method scales to room-size scenes and synthesizes plausible geometry and appearance. We visualize the generated 3D Gaussian ellipsoids and their renderings.