Gaussian Splatting Decoder for 3D-aware Generative Adversarial Networks

TL;DR

This work tackles the bottleneck of NeRF-based 3D rendering in 3D-aware GANs by introducing a Gaussian Splatting decoder that translates NeRF-derived tri-plane representations into explicit 3D Gaussian splat scenes. The decoder operates sequentially to produce color, opacity, rotation, scale, and position, enabling real-time, high-resolution rendering without a super-resolution module. By jointly fine-tuning the backbone GAN and employing a robust loss that includes perceptual and identity terms, the method achieves high visual fidelity (e.g., ID similarity up to 0.968) while delivering roughly 4× fps improvements over the original GAN renderers and preserving exportability to 3D tooling. This approach enables accurate 3D asset generation, editing, and inversion in explicit 3D space with practical performance gains for VR/AR and game pipelines.

Abstract

NeRF-based 3D-aware Generative Adversarial Networks (GANs) like EG3D or GIRAFFE have shown very high rendering quality under large representational variety. However, rendering with Neural Radiance Fields poses challenges for 3D applications: First, the significant computational demands of NeRF rendering preclude its use on low-power devices, such as mobiles and VR/AR headsets. Second, implicit representations based on neural networks are difficult to incorporate into explicit 3D scenes, such as VR environments or video games. 3D Gaussian Splatting (3DGS) overcomes these limitations by providing an explicit 3D representation that can be rendered efficiently at high frame rates. In this work, we present a novel approach that combines the high rendering quality of NeRF-based 3D-aware GANs with the flexibility and computational advantages of 3DGS. By training a decoder that maps implicit NeRF representations to explicit 3D Gaussian Splatting attributes, we can integrate the representational diversity and quality of 3D GANs into the ecosystem of 3D Gaussian Splatting for the first time. Additionally, our approach allows for a high resolution GAN inversion and real-time GAN editing with 3D Gaussian Splatting scenes. Project page: florian-barthel.github.io/gaussian_decoder

Figures (12)

• Figure 1: We propose a novel 3D Gaussian Splatting decoder that converts high quality results from pre-trained 3D-aware GANs into Gaussian Splatting scenes in real-time for efficient and high resolution rendering.
• Figure 2: Visualization of our method (orange parts are optimized). We initially clone the backbone of the 3D-aware GAN. Afterwards, we iteratively optimize the Gaussian Splatting decoder by comparing the output of the pre-trained GAN, after super-resolution, with the output of the decoder. The xyz coordinates at which the tri-plane is sampled originates from the density information of the NeRF renderer.
• Figure 3: A comparison between a parallel decoder that maps all Gaussian attributes at once to our sequential decoder, where each attribute is decoded after another using the prior information.
• Figure 4: Example renderings of the target images produced by respective 3D-aware GAN (top row) and the renderings of the decoded 3D Gaussian Splatting scene (bottom row, Ours). Additional renderings can be found in the supplementary material.
• Figure 5: Renderings for example interpolating paths, demonstrating the possibility for applying GAN editing methods.