TPA3D: Triplane Attention for Fast Text-to-3D Generation

TL;DR

This work proposes Triplane Attention for text-guided 3D generation (TPA3D), an end-to-end trainable GAN-based deep learning model for fast text- to-3D generation and generates high-quality 3D textured shapes aligned with fine-grained descriptions.

Abstract

Due to the lack of large-scale text-3D correspondence data, recent text-to-3D generation works mainly rely on utilizing 2D diffusion models for synthesizing 3D data. Since diffusion-based methods typically require significant optimization time for both training and inference, the use of GAN-based models would still be desirable for fast 3D generation. In this work, we propose Triplane Attention for text-guided 3D generation (TPA3D), an end-to-end trainable GAN-based deep learning model for fast text-to-3D generation. With only 3D shape data and their rendered 2D images observed during training, our TPA3D is designed to retrieve detailed visual descriptions for synthesizing the corresponding 3D mesh data. This is achieved by the proposed attention mechanisms on the extracted sentence and word-level text features. In our experiments, we show that TPA3D generates high-quality 3D textured shapes aligned with fine-grained descriptions, while impressive computation efficiency can be observed.

Figures (15)

• Figure 1: Overview of TPA3D for fast text-guided 3D generation. By taking sentence and word-level features $t_s$ and $t_w$ as the inputs, TPA3D utilizes generator $G$ and triplane attention (TPA) modules to predict the associated triplane features for 3D textured mesh generation, with 3D content information properly observed. Following GET3D gao2022get3d, each $G$ contains branches for geometry and texture synthesis. Note that InstructBLIP dai2023instructblip is applied to produce pseudo captions from rendered images during training, while CLIP radford2021learning extracts the resulting text features. And, $c_i$ and $f_i$ denote the sentence and word-level triplane features at each layer $i$, respectively.
• Figure 2: Design of TriPlane Attention (TPA). TPA first performs plane-wise self-attention and cross-plane attention to 3D triplane features to enforce intra-plane consistency and 3D spatial connectivity, respectively. Cross-word attention is subsequently performed to exploit word-level features for incorporating detailed information. Note that for TPA$_{\text{tex}}$, additional geometry triplane features $f^g_i$ are included to incorporate geometry information for texture generation.
• Figure 3: Qualitative comparisons with TAPS3D on (a) ShapeNet and (b) OmniObject3D. Given detailed input prompts, our TPA3D generates accurate shapes aligned to prompts, while TAPS3D only realizes general classes and simple colors.
• Figure 4: Example text-guided 3D generation results of TPA3D. We consider input prompts of "a {color} {object}" with multiple colors and sub-classes for generation. Each column stands for a different color, while each row stands for a unique sub-class: (a)"muscle car" (b)"pickup truck" (c)"sofa" (d)"office chair" (e)"scooter" (f)"dirt bike". Note that the same seeds are applied for sampling $\mathbf{z}_{\text{geo}}$ and $\mathbf{z}_{\text{tex}}$ for each row.
• Figure 5: Examples of chair manipulation by adding different detailed text descriptions. The left shows a chair generated from the input text "a wooden chair". With the same random seed for sampling $\mathbf{z}_{\text{geo}}$ and $\mathbf{z}_{\text{tex}}$, five distinct manipulations are produced by adding different detailed text descriptions.