Towards Implicit Text-Guided 3D Shape Generation

TL;DR

This paper tackles text-to-3D-shape generation with color by decoupling shape and color prediction and introducing a word-level spatial transformer to align textual descriptions with spatial regions. It combines an implicit occupancy-based decoder with cyclic consistency loss to improve text-shape fidelity and employs a style-based latent IMLE generator for diverse outputs, extended to text-guided shape manipulation via a two-way cyclic loss. The proposed three-stage framework—shape auto-encoder, text-guided generation, and diversification—achieves higher fidelity and better text-shape alignment than prior work on ShapeNet-based datasets. The work enables flexible, color-aware text-driven 3D generation and manipulation, with practical implications for rapid 3D content creation and editing.

Abstract

In this work, we explore the challenging task of generating 3D shapes from text. Beyond the existing works, we propose a new approach for text-guided 3D shape generation, capable of producing high-fidelity shapes with colors that match the given text description. This work has several technical contributions. First, we decouple the shape and color predictions for learning features in both texts and shapes, and propose the word-level spatial transformer to correlate word features from text with spatial features from shape. Also, we design a cyclic loss to encourage consistency between text and shape, and introduce the shape IMLE to diversify the generated shapes. Further, we extend the framework to enable text-guided shape manipulation. Extensive experiments on the largest existing text-shape benchmark manifest the superiority of this work. The code and the models are available at https://github.com/liuzhengzhe/Towards-Implicit Text-Guided-Shape-Generation.

Figures (22)

• Figure 1: (a) Chairs of different structures and appearances generated by our method from the same given sentence. Our method also allows text-based manipulation in color (b) and in shape (c).
• Figure 2: Overview of our text-guided shape generation framework, which has three major parts. (a) First, the shape auto-encoder $\{E,D\}$ extracts shape feature $f_s$ and color feature $f_c$ from the input 3D shape $I$. (b) We then learn to generate the 3D shape in a text-guided manner with the word-level spatial transformer (WLST) and the cyclic consistency loss $f_{cyc}$. (c) Further, we generate diversified 3D shapes from the same given text by adopting a style-based latent shape generator $G$. We only need (c) during the inference.
• Figure 3: The Word-Level Spatial Transformer architecture. $F_Q$, $F_K$, and $F_V$ are fully-connected layers, whereas $FF_1$ and $FF_2$ are feed-forward networks. The Layer Normalization ba2016layer is omitted.
• Figure 4: Visualizing the attention map $A$ for the words "metal" and "brown". Warmer colors indicate stronger correlation.
• Figure 5: The architecture of our shape-IMLE generator. Inspired by StyleGAN karras2019style, we map random noise $z$ to latent space W+ abdal2019image2stylegan to control the generator through adaptive Layer Normalization ba2016layer ($A_1$ and $A_2$) at the first and third fully-connected layers.