IRGS: Inter-Reflective Gaussian Splatting with 2D Gaussian Ray Tracing

TL;DR

This work introduces inter-reflective Gaussian splatting (IRGS) for inverse rendering, and presents an efficient optimization scheme to handle the computational demands of Monte Carlo sampling for rendering equation evaluation.

Abstract

In inverse rendering, accurately modeling visibility and indirect radiance for incident light is essential for capturing secondary effects. Due to the absence of a powerful Gaussian ray tracer, previous 3DGS-based methods have either adopted a simplified rendering equation or used learnable parameters to approximate incident light, resulting in inaccurate material and lighting estimations. To this end, we introduce inter-reflective Gaussian splatting (IRGS) for inverse rendering. To capture inter-reflection, we apply the full rendering equation without simplification and compute incident radiance on the fly using the proposed differentiable 2D Gaussian ray tracing. Additionally, we present an efficient optimization scheme to handle the computational demands of Monte Carlo sampling for rendering equation evaluation. Furthermore, we introduce a novel strategy for querying the indirect radiance of incident light when relighting the optimized scenes. Extensive experiments on multiple standard benchmarks validate the effectiveness of IRGS, demonstrating its capability to accurately model complex inter-reflection effects.

Figures (16)

• Figure 1: Global and indirect illumination in a Gaussian-based scene using our IRGS, highlighting its inter-reflective capabilities.
• Figure 2: Schematic illustration of the proposed IRGS. Starting from a set of 2D Gaussians equipped with material properties, we apply rasterization to generate albedo, roughness, position, and normal maps. We then evaluate the rendering equation using stratified sampling at the corresponding position, drawing geometry and material values from these feature maps. The radiance of incident light is decomposed into direct radiance $L_\mathrm{dir}$ from the environment map, and indirect radiance $L_\mathrm{ind}$ with visibility $V$, obtained via 2D Gaussian ray tracing.
• Figure 3: Performance of directly applying Gaussian ray tracing on a pretrained Gaussian splatting checkpoint in 3D Gaussian and 2D Gaussian cases, respectively. Our proposed 2DGRT achieves significantly less quality degradation than 3DGRT MonneLoccoz20243DGR in both quantitative metrics and visual results.
• Figure 4: Qualitative comparison of NVS, material and lighting estimation, and relighting results on the Synthetic4Relight dataset zhang2022modeling.
• Figure 5: Visualization of estimated components in incident light, including the averaged direct radiance $L_\mathrm{dir}$, indirect radiance $L_\mathrm{ind}$, visibility $V$ (ambient occlusion), incident radiance $L_\mathrm{i}$. Compared to R3DG gao2023relightable, IRGS achieves notably more realistic results, especially in estimated indirect light, due to its accurate modeling of inter-reflections.