GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space

TL;DR

GANFusion addresses the challenge of text-conditioned 3D generation using only 2D supervision by coupling an unconditional GAN-based learning stage for triplane features with a subsequent diffusion-based, text-conditioned stage trained in the GAN’s latent space. The method delivers a feed-forward text-to-3D generator that avoids test-time optimization, enabling efficient synthesis of high-quality 3D humans and related objects from natural language prompts. By generating a large 2D synthetic dataset and captioning samples to form (caption,triplane) pairs, the diffusion model learns to sample from the space of good triplane features that can be decoded into 3D geometry. The approach demonstrates competitive quality and text adherence across real and synthetic domains (FFHQ, AFHQ, DeepFashion) and shows the practical impact of leveraging 2D supervision for scalable, text-guided 3D synthesis without explicit 3D data.

Abstract

We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image or video generative models. State-of-the-art 3D generators are either trained with explicit 3D supervision and are thus limited by the volume and diversity of existing 3D data. Meanwhile, generators that can be trained with only 2D data as supervision typically produce coarser results, cannot be text-conditioned, or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast, denoising diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion, which starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN, caption them, and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects.

Figures (11)

• Figure 1: We propose GANFusion, a text-guided feed-forward 3D generator that is trained with only single-view image supervision. Unlike previous methods such as AG3D dong2023ag3d which do not enable text conditioning, GANFusion can be conditioned on text while still achieving high generation quality compared to other text conditioned generators such as RenderDiffusion anciukevivcius2023renderdiffusion. We note that GANFusion, unlike SDS-based optimization methods that use image diffusion priors poole2022dreamfusion, does not require any test-time optimization.
• Figure 2: Method Overview. We train a feed-forward text-to-3D diffusion model in two stages. First, we train an unconditional GAN-based 3D object generator like AG3D dong2023ag3d with a single-view image dataset. 3D objects are represented as triplanes and rendered with a renderer followed by an upsampler. We generate a large set of triplanes and caption them using BLIP li2022blip. The resulting (triplane, caption) dataset is then used to train a text-to-3D diffusion model, effectively distilling the GAN generator into a diffusion model, while also allowing for text conditioning. Triplanes generated by the text-to-3D model are rendered using the renderer and upsampler trained in the first stage.
• Figure 3: Qualitative comparison. We provide visual results from our method as well as the baselines. We provide random generations from AG3D dong2023ag3d which is unconditional. While random samples are of high quality, it is not straightforward to extend AG3D to enable text conditioning even with the addition of CLIP based losses during training. RenderDiffusion anciukevivcius2023renderdiffusion can follow the text prompts but lacks in terms of quality. GANFusion matches the quality of AG3D while enabling text conditioning. For AG3D and our results, we visualize the results with the 2D upsampler trained jointly with the 3D GAN generator in the first stage of our method.
• Figure 4: Qualitative results on the FFHQ dataset chan2022eg3d. We replace AG3D dong2023ag3d with EG3D chan2022eg3d as the generator in our first stage to effectively enable text-conditioning on real-world face data.
• Figure 5: Dataset examples. We show a few images and corresponding prompts from our single-view image dataset. Prompts that were initially used to generate the images are often not accurate, we refine them with our BLIP-based captioning approach.