Efficient Geometry-aware 3D Generative Adversarial Networks

TL;DR

This work tackles the challenge of generating high-quality, multi-view-consistent 3D content from 2D photographs without explicit 3D supervision. It introduces a tri-plane hybrid explicit-implicit 3D representation and a StyleGAN2-based generator that decouples feature generation from neural rendering, enabling real-time, high-resolution 3D-aware synthesis and accurate geometry. A dual discriminator and pose conditioning regularize the training to maintain view consistency while modeling pose-correlated attributes, achieving state-of-the-art results on FFHQ and AFHQ Cats. The approach leverages 2D CNN generators for efficiency and demonstrates practical benefits in style mixing and single-view 3D reconstruction, with potential impacts on rapid 3D content creation and reconstruction from 2D data.

Abstract

Unsupervised generation of high-quality multi-view-consistent images and 3D shapes using only collections of single-view 2D photographs has been a long-standing challenge. Existing 3D GANs are either compute-intensive or make approximations that are not 3D-consistent; the former limits quality and resolution of the generated images and the latter adversely affects multi-view consistency and shape quality. In this work, we improve the computational efficiency and image quality of 3D GANs without overly relying on these approximations. We introduce an expressive hybrid explicit-implicit network architecture that, together with other design choices, synthesizes not only high-resolution multi-view-consistent images in real time but also produces high-quality 3D geometry. By decoupling feature generation and neural rendering, our framework is able to leverage state-of-the-art 2D CNN generators, such as StyleGAN2, and inherit their efficiency and expressiveness. We demonstrate state-of-the-art 3D-aware synthesis with FFHQ and AFHQ Cats, among other experiments.

Figures (9)

• Figure 1: Our 3D GAN enables synthesis of scenes, producing high-quality, multi-view-consistent renderings and detailed geometry. Our approach trains from a collection of 2D images without target-specific shape priors, ground truth 3D scans, or multi-view supervision. Please see the accompanying video for more results.
• Figure 2: Neural implicit representations use fully connected layers (FC) with positional encoding (PE) to represent a scene, which can be slow to query (a). Explicit voxel grids or hybrid variants using small implicit decoders are fast to query, but scale poorly with resolution (b). Our hybrid explicit--implicit tri-plane representation (c) is fast and scales efficiently with resolution, enabling greater detail for equal capacity.
• Figure 3: A synthesized view of the multi-view Family scene, comparing a fully implicit Mip-NeRF representation (left), a dense voxel grid (center), and our tri-plane representation (right). Even though neither voxels nor tri-planes model view-dependent effects, they achieve high quality.
• Figure 4: Our 3D GAN framework comprises several parts: a pose-conditioned StyleGAN2-based feature generator and mapping network, a tri-plane 3D representation with a lightweight feature decoder, a neural volume renderer, a super-resolution module, and a pose-conditioned StyleGAN2 discriminator with dual discrimination. This architecture elegantly decouples feature generation and neural rendering, allowing the use of a powerful StyleGAN2 generator for 3D scene generalization. Moreover, the lightweight 3D tri-plane representation is both expressive and efficient in enabling high-quality 3D-aware view synthesis in real-time.
• Figure 5: Dual discrimination ensures that the raw neural rendering $I_{RGB}$ and super-resolved output $I^+_{RGB}$ maintain consistency, enabling high-resolution and multi-view-consistent rendering.