GS-IR: 3D Gaussian Splatting for Inverse Rendering

TL;DR

GS-IR introduces a forward-mapping 3D Gaussian Splatting pipeline for inverse rendering that jointly recovers geometry, materials, and unknown illumination from multi-view images. It tackles key challenges with depth-derivation based normal regularization and baking-based occlusion to model indirect lighting, enabling efficient physically-based relighting and material editing. The method achieves state-of-the-art or competitive results on synthetic and real datasets while significantly speeding up training relative to baselines. Limitations include SH-based indirect illumination being low-frequency and incomplete modeling of specular indirect lighting, suggesting directions like screen-space GI for future work.

Abstract

We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) that leverages forward mapping volume rendering to achieve photorealistic novel view synthesis and relighting results. Unlike previous works that use implicit neural representations and volume rendering (e.g. NeRF), which suffer from low expressive power and high computational complexity, we extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. There are two main problems when introducing GS to inverse rendering: 1) GS does not support producing plausible normal natively; 2) forward mapping (e.g. rasterization and splatting) cannot trace the occlusion like backward mapping (e.g. ray tracing). To address these challenges, our GS-IR proposes an efficient optimization scheme that incorporates a depth-derivation-based regularization for normal estimation and a baking-based occlusion to model indirect lighting. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering. We demonstrate the superiority of our method over baseline methods through qualitative and quantitative evaluations on various challenging scenes.

Figures (13)

• Figure 1: Given multi-view captured images of a complex scene, we propose GS-IR (3D Gaussian Splatting for Inverse Rendering), which utilizes 3D Gaussian and forward mapping splatting to recover high-quality physical properties (e.g., normal, material, illumination). This enables us to perform relighting and material editing, resulting in outstanding inverse rendering results. Better viewed on screen with zoom in, especially the remarkable material decomposition and normal reconstruction of bicycle axle.
• Figure 2: GS-IR Pipeline. We propose a novel GS-based inverse rendering framework, called GS-IR, to reconstruct scene geometry, materials, and unknown natural illumination from multi-view captured images. Our GS-IR consists of three well-designed stage strategies using 3D Gaussian and differentiable forward mapping splatting to achieve physical-based rendering. In our approach, the Gaussian stores not only the basic 3DGS information but also the normal and material properties, enhancing its capabilities for inverse rendering tasks.
• Figure 3: Depth Illustration. By considering the depth as a linear interpolation of the distances from 3D Gaussians to the image plane, and ensuring it lies between the minimum and maximum distance, our method could produce accurate depth.
• Figure 4: Baking. We employ the spherical harmonics (SH) architecture to bake occlusion volumes for modeling indirect illumination. For each grid of occlusion volumes, we initially use 3D Gaussians to compute the depth cubemap by performing six forward mapping splatting passes. Next, we convert the depth cubemap into a binary occlusion cubemap based on a distance threshold. Finally, the occlusion cubemap is baked as SH coefficients, enabling efficient interpolation of the occlusion cubemap at any point within the scene.
• Figure 5: Qualitative comparison on TensoIR Synthetic. We visualize the estimated normal, albedo, and rendering results of our GS-IR and baseline methods on two scenes. By utilizing the efficient 3D Gaussian representation and a robust tile-based rasterizer, GS-IR achieves rapid convergence and supports real-time rendering. This performance advantage underscores the effectiveness of our method in addressing complex inverse rendering tasks, thereby surpassing existing state-of-the-art approaches. (For albedo reconstruction results, we follow NeRFactor zhang2021nerfactor and scale each RGB channel by a global scalar.)