Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation

TL;DR

The paper tackles cross-modal 3D shape generation from 2D images and textual descriptions by introducing an alignment-before-generation paradigm. It proposes Shape-Image-Text-Aligned VAE (SITA-VAE) to embed 3D shapes into a latent space aligned with images and texts via CLIP-based contrastive losses, and Aligned Shape Latent Diffusion Model (ASLDM) to map these conditions to shape latents for diffusion-based generation. The framework yields high-quality, diverse 3D shapes with strong semantic fidelity, validated on ShapeNet and a Cartoon Monster dataset using IoU, SI-S, ST-S, P-FID, and P-IS metrics, and ablations highlight the importance of the aligned latent space and CLIP guidance. This alignment-space approach reduces the distribution gap between modalities, enabling efficient, scalable, and semantically coherent cross-modal 3D generation without per-scene optimization.

Abstract

We present a novel alignment-before-generation approach to tackle the challenging task of generating general 3D shapes based on 2D images or texts. Directly learning a conditional generative model from images or texts to 3D shapes is prone to producing inconsistent results with the conditions because 3D shapes have an additional dimension whose distribution significantly differs from that of 2D images and texts. To bridge the domain gap among the three modalities and facilitate multi-modal-conditioned 3D shape generation, we explore representing 3D shapes in a shape-image-text-aligned space. Our framework comprises two models: a Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and a conditional Aligned Shape Latent Diffusion Model (ASLDM). The former model encodes the 3D shapes into the shape latent space aligned to the image and text and reconstructs the fine-grained 3D neural fields corresponding to given shape embeddings via the transformer-based decoder. The latter model learns a probabilistic mapping function from the image or text space to the latent shape space. Our extensive experiments demonstrate that our proposed approach can generate higher-quality and more diverse 3D shapes that better semantically conform to the visual or textural conditional inputs, validating the effectiveness of the shape-image-text-aligned space for cross-modality 3D shape generation.

Figures (11)

• Figure 1: Visualization of the 3D shape produced by our framework, which splits into triplets with a conditional input on the left, a normal map in the middle, and a triangle mesh on the right. The generated 3D shapes semantically conform to the visual or textural conditional inputs.
• Figure 2: Alignment-before-generation pipeline. Our method contains two models: the Shape-Image-Text-Aligned Variational Auto-Encoder (SITA-VAE) and the Aligned Shape Latent Diffusion Model (ASLDM). The SITA-VAE consists of four modules: an image encoder, a text encoder, a 3D shape encoder, and a 3D shape decoder. Encoders encode inputs pair into an aligned space, and the 3D shape decoder reconstructs 3D shapes given embeddings from the aligned space. The ASLDM maps the image or text condition to the aligned shape latent space for sampling a high-quality 3D shape embedding, which latterly reconstructed to high-fidelity 3D shapes by the 3D shape decoder.
• Figure 3: Visual results for image-conditioned generation comparison. The figure shows that 3DILG zhang20223dilg generates over-smooth surfaces and lacks details of shapes, whereas 3DS2V zhang20233dshape2vecset generates few details with noisy and discontinuous surfaces of shapes. In contrast to baselines, our method produces smooth surfaces and portrays shape details. Please zoom in for more visual details.
• Figure 4: Visual results for text-conditioned generation comparison. In the first two rows, we test the model with abstract texts, and the result shows that only our model could generate a 3D shape that conforms to the target text with a smooth surface and fine details. The last two rows show the result given texts containing detailed descriptions, which further shows that our model could capture the global conditional information and the local information for generating high-fidelity 3D shapes. Keywords are highlighted in red; please zoom in for more visual details.
• Figure 5: Ablation study the effectiveness of training generative model in the aligned space. This figure illustrates visual comparisons for ablation studies on the effectiveness of training the generative model in the aligned space. Compared with the lower samples based on the conditional texts, the upper samples are closer to the conditions semantically, which indicates the effectiveness of the training generative model in the aligned space.