GaussianShader: 3D Gaussian Splatting with Shading Functions for Reflective Surfaces

TL;DR

GaussianShader addresses the challenge of rendering reflective surfaces in neural 3D Gaussian representations by introducing a shading function for Gaussians and a novel normal estimation approach. The method combines a diffuse term, direct specular lighting, and a residual SH-based term, all fed through environment-map lighting and a GGX-based specular integration, with a normal predicted from the Gaussian's shortest axis plus a learnable residual and a normal-geometry consistency loss. This yields high-quality specular rendering while preserving the real-time efficiency of 3D Gaussian Splatting and significantly reducing optimization time versus Ref-NeRF/ENVIDR. The work enables robust, interactive rendering of scenes with complex materials and reflections, broadening the applicability of neural rendering to practical reflective objects.

Abstract

The advent of neural 3D Gaussians has recently brought about a revolution in the field of neural rendering, facilitating the generation of high-quality renderings at real-time speeds. However, the explicit and discrete representation encounters challenges when applied to scenes featuring reflective surfaces. In this paper, we present GaussianShader, a novel method that applies a simplified shading function on 3D Gaussians to enhance the neural rendering in scenes with reflective surfaces while preserving the training and rendering efficiency. The main challenge in applying the shading function lies in the accurate normal estimation on discrete 3D Gaussians. Specifically, we proposed a novel normal estimation framework based on the shortest axis directions of 3D Gaussians with a delicately designed loss to make the consistency between the normals and the geometries of Gaussian spheres. Experiments show that GaussianShader strikes a commendable balance between efficiency and visual quality. Our method surpasses Gaussian Splatting in PSNR on specular object datasets, exhibiting an improvement of 1.57dB. When compared to prior works handling reflective surfaces, such as Ref-NeRF, our optimization time is significantly accelerated (23h vs. 0.58h). Please click on our project website to see more results.

Figures (11)

• Figure 1: GaussianShader maintains real-time rendering speed and renders high-fidelity images for both general and reflective surfaces. Ref-NeRFverbin2022ref and ENVIDRliang2023envidr attempt to handle reflective surfaces, but they suffer from quite time-consuming optimization and slow rendering speed. 3D Gaussian splatting kerbl20233d keeps high efficiency but cannot handle such reflective surfaces.
• Figure 2: GaussianShader initiates with the neural 3D Gaussian spheres that integrate both conventional attributes and the newly introduced shading attributes to accurately capture view-dependent appearances. We incorporate a differentiable environment lighting map to simulate realistic lighting. The end-to-end training leads to a model that reconstructs both reflective and diffuse surfaces, achieving high material and lighting fidelity.
• Figure 3: Normal Distribution Function D in Eq. \ref{['eq_light']} is determined by the roughness $\rho$ and reflective direction $\mathbf{r}$. Surface with small $\rho$ has a smaller specular lobe and that with large $\rho$ has a larger specular lobe.
• Figure 4: The geometric evolving process of a 3D Gaussian sphere in the optimization, which gradually becomes planar.
• Figure 5: Visualization of the relationship between shortes axis $\mathbf{v}$, normal residual $\Delta \mathbf{n}$, normal $\mathbf{n}$ and depth-grad normal $\hat{\mathbf{n}}$. The supervision $\mathcal{L}_{normal}$ enforces the normal-geometry consistency.