GUS-IR: Gaussian Splatting with Unified Shading for Inverse Rendering

TL;DR

This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading, effectiveness in handling complex materials and proposes a unified shading solution that combines the advantages of both techniques for better decomposition.

Abstract

Recovering the intrinsic physical attributes of a scene from images, generally termed as the inverse rendering problem, has been a central and challenging task in computer vision and computer graphics. In this paper, we present GUS-IR, a novel framework designed to address the inverse rendering problem for complicated scenes featuring rough and glossy surfaces. This paper starts by analyzing and comparing two prominent shading techniques popularly used for inverse rendering, forward shading and deferred shading, effectiveness in handling complex materials. More importantly, we propose a unified shading solution that combines the advantages of both techniques for better decomposition. In addition, we analyze the normal modeling in 3D Gaussian Splatting (3DGS) and utilize the shortest axis as normal for each particle in GUS-IR, along with a depth-related regularization, resulting in improved geometric representation and better shape reconstruction. Furthermore, we enhance the probe-based baking scheme proposed by GS-IR to achieve more accurate ambient occlusion modeling to better handle indirect illumination. Extensive experiments have demonstrated the superior performance of GUS-IR in achieving precise intrinsic decomposition and geometric representation, supporting many downstream tasks (such as relighting, retouching) in computer vision, graphics, and extended reality.

Figures (11)

• Figure 1: Given multi-view captured images of a complex scene featuring rough and glossy surfaces, we propose GUS-IR (Gaussian Splatting with Unified Shading for Inverse Rendering), which utilizes 3D Gaussians to recover high-quality physical properties (e.g., normal, material, illumination) under unknown illumination. This enables us to perform advanced applications (e.g. relighting), resulting in outstanding inverse rendering results. Better viewed on screen with zoom-in, we successfully recovered the glossy surfaces of the marble balls placed in the center of the garden.
• Figure 2: Overview of GUS-IR. During optimization, GUS-IR simultaneously conducts forward and deferred shading schemes and supervises the rendering results produced by both schemes. We use the shortest axis towards the view as the particle's normal for forward shading and render a normal map for deferred shading.
• Figure 3: Difference between forward and deferred shading schemes. $\Sigma$ denotes the accumulation in the volume rendering. $\bm{n}_k, \Omega_k, \Phi_k$ are the normal, upper hemisphere, and BRDF lobe of the $k$-th particle, respectively. The table bell examples show that deferred shading can better capture and reconstruct glossy details than forward shading.
• Figure 4: Overview of caching occlusion in the baking stage and recovering occlusion in the decomposition from volumes. We employ the dense structure to bake occlusion volumes for modeling ambient occlusion. For each volume $\bm{v}^\text{O}$, we run six passes to render the depth cubemap $\{\bm{D}^{\bm{v}^\text{O}}_i\}^6_{i=1}$ and convert the cubemap into a binary occlusion cubemap $\{\bm{O}^{\bm{v}^\text{O}}_i\}^6_{i=1}$ through a manually set distance threshold. Then we flatten the binary occlusion cubemap and store it in a dense byte array. Given a surface point $\bm{x}$ with normal $\bm{n}$, we conduct normal-aware masked-trilinear interpolation to get the occlusion cubemap, averaging the uniform sampling occlusion as the ambient occlusion $O(\bm{x})$.
• Figure 5: Normal estimation comparisons on TensoIR jin2023tensoir and Shiny Blender verbin2022ref datasets. We exemplify the normal estimation results of our GUS-IR and other cutting-edge methods.