GIR: 3D Gaussian Inverse Rendering for Relightable Scene Factorization

TL;DR

Problem: factorizing geometry, materials, and lighting from multi-view images is ill-posed. Approach: extend 3D Gaussian Splatting with inverse rendering (GIR) using normals from the shortest eigenvector, voxel-based indirect illumination tracing, and a learnable high-resolution illumination map. Contributions: directional masking for normals, efficient indirect illumination disentangling, and FCN-based illumination learning enabling high-quality relighting and NVS in real time. Findings: GIR achieves state-of-the-art results among recent inverse-rendering methods and supports interactive material editing and relighting in real time.

Abstract

This paper presents a 3D Gaussian Inverse Rendering (GIR) method, employing 3D Gaussian representations to effectively factorize the scene into material properties, light, and geometry. The key contributions lie in three-fold. We compute the normal of each 3D Gaussian using the shortest eigenvector, with a directional masking scheme forcing accurate normal estimation without external supervision. We adopt an efficient voxel-based indirect illumination tracing scheme that stores direction-aware outgoing radiance in each 3D Gaussian to disentangle secondary illumination for approximating multi-bounce light transport. To further enhance the illumination disentanglement, we represent a high-resolution environmental map with a learnable low-resolution map and a lightweight, fully convolutional network. Our method achieves state-of-the-art performance in both relighting and novel view synthesis tasks among the recently proposed inverse rendering methods while achieving real-time rendering. This substantiates our proposed method's efficacy and broad applicability, highlighting its potential as an influential tool in various real-time interactive graphics applications such as material editing and relighting. The code will be released at https://github.com/guduxiaolang/GIR.

Figures (12)

• Figure 1: Our proposed 3D GIR offers the capability to reconstruct high-quality scenes from multi-view images, incorporating factored properties associated with physically-based rendering, thereby empowering users to interactively edit these elements or conduct relighting under novel lighting conditions.
• Figure 2: The outline of the proposed GIR is illustrated on the left. It optimizes three material factors, two light factors, and four geometry factors to represent a scene. An FCN is employed to learn high-quality environmental illumination by optimizing the projection of 2D-environmental embedding onto a high-resolution environmental map. Indirect Light Estimation illustrates how a 3D Gaussian is influenced by indirect illumination. Directional Masking illustrates calculating the mask for a 3D Gaussian's color, utilizing the unraveled normal $n$. The details are explained in Sec. \ref{['sec:method:normal_estimation']}.
• Figure 3: Qualitative comparison of NVS on the Shiny Blender (first three rows) and Glossy Blender (last three rows) datasets.
• Figure 4: Qualitative comparison of relighting on the TensoIR dataset. The first column shows four new environmental lights and two reference images.
• Figure 5: Qualitative comparison of albedo. Left: TensoIR dataset. Right: Shiny Blender dataset. Since the TensoIR dataset does not provide GT albedo, we use GT images for reference. Note that GS Shader is not included in the comparison due to the absence of albedo.