RefGaussian: Disentangling Reflections from 3D Gaussian Splatting for Realistic Rendering

TL;DR

RefGaussian tackles the challenge of modeling reflections in 3D Gaussian Splatting by disentangling a scene into transmitted and reflected components using reflection-SH, reflection opacity, and reflection confidence. The method extends the Gaussian representation with a compact set of reflection-related parameters and couples it with bilateral smoothness priors, enabling mask-free, joint rendering of both components via a single, efficient framework. Empirical results on reflective real-world scenes show improved view synthesis and depth estimation over NeRF- and 3D-GS-based baselines, while preserving fast rendering and allowing reflection-level scene editing. This approach offers a practical pathway to physically coherent reflections in real-time neural rendering and opens avenues for interactive scene manipulation.

Abstract

3D Gaussian Splatting (3D-GS) has made a notable advancement in the field of neural rendering, 3D scene reconstruction, and novel view synthesis. Nevertheless, 3D-GS encounters the main challenge when it comes to accurately representing physical reflections, especially in the case of total reflection and semi-reflection that are commonly found in real-world scenes. This limitation causes reflections to be mistakenly treated as independent elements with physical presence, leading to imprecise reconstructions. Herein, to tackle this challenge, we propose RefGaussian to disentangle reflections from 3D-GS for realistically modeling reflections. Specifically, we propose to split a scene into transmitted and reflected components and represent these components using two Spherical Harmonics (SH). Given that this decomposition is not fully determined, we employ local regularization techniques to ensure local smoothness for both the transmitted and reflected components, thereby achieving more plausible decomposition outcomes than 3D-GS. Experimental results demonstrate that our approach achieves superior novel view synthesis and accurate depth estimation outcomes. Furthermore, it enables the utilization of scene editing applications, ensuring both high-quality results and physical coherence.

Figures (7)

• Figure 1: Novel view synthesis with RefGaussian, fulfilling better reflections modeling than original 3D-GS.
• Figure 2: The overview of our proposed RefGaussian. The RefGaussian framework is specifically designed to accurately model general reflections within a scene by effectively decomposing it into two distinct components: the transmitted component and the reflected component. RefGaussian eliminates the requirement for additional 3D Gaussians, resulting in a more efficient and realistic rendering process. By incorporating bilateral smoothness and reflection map smoothness, the framework enables effective scene decomposition. Moreover, both depth variations and color differences are taken into account and collaboratively optimized to further enhance the overall rendering quality.
• Figure 3: Visual comparisons between NeRF, NeRF-D, NeRFReN, 3D-GS and our method. Our method presents a more detailed and realistic rendering than 3D-GS in all cases. Compared to NeRF-based methods, our methods show comparable results in scenes with semi-reflections and exceed them in scenes containing specular reflections.
• Figure 4: Detailed visual comparisons between 3D-GS and our method. For a comprehensive comparison, we further show visual results of 3D-GS and our method with zoom-in details on RFFR scenes bookcase and mirror, and more general scenes like room from the Mip-NeRF360 dataset and truck from Tanks&Temples.
• Figure 5: Illustration of scene disentanglement. The final rendered images (a) combine transmitted components (b) with the product of reflected components (c) and the reflection fraction map (d).