3D Gaussian Inverse Rendering with Approximated Global Illumination

TL;DR

This work enables realistic, editable rendering for Gaussian Splatting by introducing a screen-space ray tracing pipeline that estimates one-bounce global illumination directly from G-buffers. By augmenting each Gaussian with Disney BRDF parameters and performing deferred shading, the method captures direct illumination, while a Monte-Carlo screen-space pass accounts for indirect lighting without full scene traversal. The approach uses depth-based normals for stable shading, a split-sum BRDF approximation with a learnable environment map, and a lightweight SSR step to maintain real-time performance. Extensive qualitative and quantitative evaluations on underground garage and campus scenes demonstrate faithful reconstructions, intuitive editing (object insertion, relighting, material changes), and competitive rendering speed, with limitations noted in non-differentiable SSR and distant outdoor lighting scenarios.

Abstract

3D Gaussian Splatting shows great potential in reconstructing photo-realistic 3D scenes. However, these methods typically bake illumination into their representations, limiting their use for physically-based rendering and scene editing. Although recent inverse rendering approaches aim to decompose scenes into material and lighting components, they often rely on simplifying assumptions that fail when editing. We present a novel approach that enables efficient global illumination for 3D Gaussians Splatting through screen-space ray tracing. Our key insight is that a substantial amount of indirect light can be traced back to surfaces visible within the current view frustum. Leveraging this observation, we augment the direct shading computed by 3D Gaussians with Monte-Carlo screen-space ray-tracing to capture one-bounce indirect illumination. In this way, our method enables realistic global illumination without sacrificing the computational efficiency and editability benefits of 3D Gaussians. Through experiments, we show that the screen-space approximation we utilize allows for indirect illumination and supports real-time rendering and editing. Code, data, and models will be made available at our project page: https://wuzirui.github.io/gs-ssr.

Figures (12)

• Figure 1: Overview: (a) Our inverse rendering pipeline recovers geometry and material properties from 3D captures, visualized through normal (iii) and roughness (iv) maps. The decomposition enables various editing capabilities: object insertion, material editing, and relighting. Our screen-space ray tracing technique ensures physically plausible reflections (visualized through corresponding point pairs).
• Figure 2: Pipeline: (a) Our method extends standard Gaussian splatting with material intrinsic properties. We first rasterize the Gaussian primitives into G-buffers containing both geometric and material properties of the current rendering frame; (b) We perform deferred shading on the alpha-composited G-buffers with direct environment lights from a learnable cubemap. (Sec. \ref{['sec:deferred-shading']}); (c) We approximate one-bounce indirect lights through screen-space ray tracing and compose the final rendered RGB through Monte-Carlo integration. (Sec.\ref{['sec:ssr']}).
• Figure 3: (a) While alpha-composited normals work well from the original camera view (top), they can produce incorrect estimates in novel views (bottom) when rays intersect with previously occluded surfaces. (b) Qualitative comparison showing that depth-based normals maintain consistency across both training and novel views, while rendered normals exhibit artifacts in novel views.
• Figure 4: Screen-space ray tracing process. For each pixel, we march along the sampled reflection ray step by step and compare the ray's depth with the scene depth buffer at each step. An intersection is detected when the ray depth transitions from being in front of to behind the depth buffer values, indicating the ray has intersected with scene geometry.
• Figure 5: Qualitative comparisons. We show the rendering results for each method. The regions highlighted in yellow boxes show reflections on the ground where baseline methods struggle to capture dramatically dynamic indirect illumination effects; Green boxes show blurry artifacts.