Ref-GS: Directional Factorization for 2D Gaussian Splatting

TL;DR

Ref-GS builds upon the deferred rendering of Gaussian splatting and applies directional encoding to the deferred-rendered surface, effectively reducing the ambiguity between orientation and viewing angle and achieves superior photorealistic rendering for a range of open-world scenes while also accurately recovering geometry.

Abstract

In this paper, we introduce Ref-GS, a novel approach for directional light factorization in 2D Gaussian splatting, which enables photorealistic view-dependent appearance rendering and precise geometry recovery. Ref-GS builds upon the deferred rendering of Gaussian splatting and applies directional encoding to the deferred-rendered surface, effectively reducing the ambiguity between orientation and viewing angle. Next, we introduce a spherical Mip-grid to capture varying levels of surface roughness, enabling roughness-aware Gaussian shading. Additionally, we propose a simple yet efficient geometry-lighting factorization that connects geometry and lighting via the vector outer product, significantly reducing renderer overhead when integrating volumetric attributes. Our method achieves superior photorealistic rendering for a range of open-world scenes while also accurately recovering geometry.

Figures (10)

• Figure 1: Our Ref-GS method generates photo-realistic renderings with view-dependent effects while also enabling accurate geometry recovery. The top row shows a comparison of renderings for a scene with specular reflections, along with the recovered normals and mesh. The bottom row demonstrates our successful reconstruction of the geometries of the 'reflective table center' in the 'Garden' scene barron2022mipnerf360 and the 'windshield' in the 'Truck' scene knapitsch2017tanks, which existing methods typically fail to handle.
• Figure 2: Overview of Ref-GS. From left to right: the geometry pass renders the scene properties, including appearance feature $\mathbf{K}$, roughness map $\mathbf{M}$, and normal map $\mathbf{N}$, into buffers via deferred rendering, the lighting pass projects the reflected direction $\omega_r$ onto spherical coordinates $(\theta, \phi)$ and featurized by $\operatorname{Sph-Mip}$ encoding for modeling far-field lighting, finally the rendering pass use tensor factorization $\mathbf{s} \circ \mathbf{k}$ to obtain spatially varying view-dependent effects and color each pixel $(u, v)$.
• Figure 3: Comparison of directional query in Gaussian Splatting. a) The original 3DGS kerbl3Dgaussians and 2DGS Huang2DGS2024 methods query each primitive's SH coefficient using the viewing direction, then accumulate view-dependent radiance as the ray color. b) Ref-NeRF verbin2022refnerf and recent GaussianShader jiang2024gaussianshader utilize the reflection direction transformed by both the viewing and normal directions as the directional query. c) We introduce Gaussian deferred shading by first integrating the SH coefficient and normal as a surface point, then evaluating its view-dependent color.
• Figure 4: Visualized estimated surface normal results synthetic datasetsverbin2022refnerfliu2023nero. Compared to existing Gaussian-based methods, our method has more accurate surface reconstruction for shiny objects with inter-reflections, as depicted for this 'Toaster' and 'Bell' scenes.
• Figure 5: Qualitative comparisons of test-set views of real-world scenes. Notice the high-frequency reflections rendered by our model, including sharp details of the tree branches and buildings reflected in the sphere.