3D Gaussian Splatting with Deferred Reflection

TL;DR

This work tackles the challenge of rendering specular reflection in 3D Gaussian Splatting by introducing a deferred shading approach that decouples base shading from reflections. It employs a two-pass pipeline: a Gaussian splatting pass to bake per-Gaussian normals and reflection strength into screen-space maps, and a per-pixel shading pass that queries a learnable environment map to produce the final color, with a training regime that enables normal propagation across Gaussians via normal propagation and color sabotage. The method achieves state-of-the-art PSNR and more accurate normal and environment-map estimates on synthetic and real datasets, while maintaining frame rates close to vanilla 3D Gaussian Splatting and requiring fewer Gaussians. This approach advances practical high-quality specular rendering in differentiable Gaussian-based radiance fields and offers a foundation for future extensions to richer reflection models and lighting decompositions.

Abstract

The advent of neural and Gaussian-based radiance field methods have achieved great success in the field of novel view synthesis. However, specular reflection remains non-trivial, as the high frequency radiance field is notoriously difficult to fit stably and accurately. We present a deferred shading method to effectively render specular reflection with Gaussian splatting. The key challenge comes from the environment map reflection model, which requires accurate surface normal while simultaneously bottlenecks normal estimation with discontinuous gradients. We leverage the per-pixel reflection gradients generated by deferred shading to bridge the optimization process of neighboring Gaussians, allowing nearly correct normal estimations to gradually propagate and eventually spread over all reflective objects. Our method significantly outperforms state-of-the-art techniques and concurrent work in synthesizing high-quality specular reflection effects, demonstrating a consistent improvement of peak signal-to-noise ratio (PSNR) for both synthetic and real-world scenes, while running at a frame rate almost identical to vanilla Gaussian splatting.

Figures (11)

• Figure 1: Our rendering pipeline. A Gaussian splatting pass is first performed to bake reflection strength, normal, and base color to screen space maps. In the following shading pass, for each pixel, we use the normal map to compute a reflection direction and query an environment map for a reflected color. The reflection strength is then used to blend base color and reflection color into the final result. An image loss is used to back-propagate gradients. Note that there exist many gradient propagation paths. Here we only illustrate the gradient flow most relevant to reflection fitting.
• Figure 2: The propagation of Gaussian normal and reflection strength at various training steps.
• Figure 3: Quality comparison between forward and deferred designs. From left to right: teapot, bell, tbell.
• Figure 4: Normal maps at various training steps with one algorithm component disabled.
• Figure 5: Decomposition results of our method. From top to bottom: ball, potion, tbell.