3D-aware Image Synthesis via Learning Structural and Textural Representations

TL;DR

VolumeGAN tackles 3D-aware image synthesis from unstructured 2D data by addressing NeRF-based generators' limitations—local receptive fields and expensive volume rendering—through explicit structural and textural representations. It introduces a learnable 3D feature volume as the structural component and a 2D feature map as the textural component, bridged by a NeRF-like feature field and rendered by a 1x1 convolutional neural renderer, enabling independent control of shape and appearance. The approach achieves state-of-the-art results on multiple real and synthetic datasets, with significant improvements in FID (e.g., FFHQ from 36.7 to 9.1) and better multi-view consistency and pose control, while demonstrating natural disentanglement between structure and texture. While offering substantial benefits for scalable, high-quality 3D-aware synthesis, the work notes limitations in mesh smoothness and acknowledges potential risks related to misuse, suggesting future work on refining geometry and safety considerations.

Abstract

Making generative models 3D-aware bridges the 2D image space and the 3D physical world yet remains challenging. Recent attempts equip a Generative Adversarial Network (GAN) with a Neural Radiance Field (NeRF), which maps 3D coordinates to pixel values, as a 3D prior. However, the implicit function in NeRF has a very local receptive field, making the generator hard to become aware of the global structure. Meanwhile, NeRF is built on volume rendering which can be too costly to produce high-resolution results, increasing the optimization difficulty. To alleviate these two problems, we propose a novel framework, termed as VolumeGAN, for high-fidelity 3D-aware image synthesis, through explicitly learning a structural representation and a textural representation. We first learn a feature volume to represent the underlying structure, which is then converted to a feature field using a NeRF-like model. The feature field is further accumulated into a 2D feature map as the textural representation, followed by a neural renderer for appearance synthesis. Such a design enables independent control of the shape and the appearance. Extensive experiments on a wide range of datasets show that our approach achieves sufficiently higher image quality and better 3D control than the previous methods.

Figures (7)

• Figure 1: Images of faces and cars synthesized by VolumeGAN, which enables the control of viewpoint, structure, and texture.
• Figure 2: Framework of the proposed VolumeGAN. We first learn a feature volume, starting from a learnable spatial template, as the structural representation. Given the camera pose $\xi$, we sample points along a camera ray and query the coordinate descriptor of each point from the feature volume via trilinear interpolation. The resulting coordinate descriptors, concatenated with the raw 3D coordinates, are then converted to a generative feature field and further accumulated as a 2D feature map. Such a feature map is regarded as the textural representation, which guides the rendering of the appearance of the output synthesis with the help of a neural renderer.
• Figure 3: Qualitative comparison between our VolumeGAN and existing alternatives on FFHQ stylegan, CompCars compcars, and LSUN bedroom lsun datasets. All images are in $256\times 256$ resolution.
• Figure 4: Synthesized results with the front camera view by $\pi$-GAN pigan and our VolumeGAN, where the faces proposed by VolumeGAN are more consistent to the given view, suggesting a better 3D controllability.
• Figure 5: Synthesized results by exchanging the structural and the textural latent codes.