Compress3D: a Compressed Latent Space for 3D Generation from a Single Image

TL;DR

Compress3D introduces a compact triplane latent space learned by a trained Triplane AutoEncoder to capture 3D geometry and texture from colored point clouds. A 3D-aware cross-attention mechanism enriches the latent representation, and a diffusion prior models shape embedding conditioned on image embedding, enabling a Triplane Diffusion Model conditioned on both image and shape information to synthesize high-quality 3D assets from a single image. The method demonstrates superior FID and CLIP scores with significantly reduced training data and time compared to state-of-the-art approaches, achieving fast inference on standard GPUs. This work advances single-view 3D generation by combining efficient latent compression with dual-condition diffusion, enabling practical creation of 3D content from a single image.

Abstract

3D generation has witnessed significant advancements, yet efficiently producing high-quality 3D assets from a single image remains challenging. In this paper, we present a triplane autoencoder, which encodes 3D models into a compact triplane latent space to effectively compress both the 3D geometry and texture information. Within the autoencoder framework, we introduce a 3D-aware cross-attention mechanism, which utilizes low-resolution latent representations to query features from a high-resolution 3D feature volume, thereby enhancing the representation capacity of the latent space. Subsequently, we train a diffusion model on this refined latent space. In contrast to solely relying on image embedding for 3D generation, our proposed method advocates for the simultaneous utilization of both image embedding and shape embedding as conditions. Specifically, the shape embedding is estimated via a diffusion prior model conditioned on the image embedding. Through comprehensive experiments, we demonstrate that our method outperforms state-of-the-art algorithms, achieving superior performance while requiring less training data and time. Our approach enables the generation of high-quality 3D assets in merely 7 seconds on a single A100 GPU.

Figures (7)

• Figure 1: Given a single-view image, our method can generate high-quality 3D Models.
• Figure 2: Method overview. Compress3D mainly contains 3 components. (a) Triplane AutoEncoder: Triplane Encoder encodes color point cloud on a low-resolution triplane latent space. Then we use a Triplane Decoder to decode 3D model from a triplane latent. (b) Triplane Diffusion Model: we use shape embedding and image embedding as conditions to generate triplane latent. (c) Diffusion Prior Model: generate shape embedding conditioned on the image embedding.
• Figure 3: Triplane Encoder. TriConv is the 3D-aware convolution proposed in wang2023rodin.
• Figure 4: 3D-aware cross attention. We use each point feature on the triplane to query the corresponding cube region (red) of feature volume. In addition, we add a position embedding to the volume feature.
• Figure 5: Triplane Decoder.