Phys3DGS: Physically-based 3D Gaussian Splatting for Inverse Rendering

TL;DR

A novel two-step training approach is proposed which exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and applies novel regularization methods to better exploit the mesh and gives better rendering quality while offering real-time rendering.

Abstract

We propose two novel ideas (adoption of deferred rendering and mesh-based representation) to improve the quality of 3D Gaussian splatting (3DGS) based inverse rendering. We first report a problem incurred by hidden Gaussians, where Gaussians beneath the surface adversely affect the pixel color in the volume rendering adopted by the existing methods. In order to resolve the problem, we propose applying deferred rendering and report new problems incurred in a naive application of deferred rendering to the existing 3DGS-based inverse rendering. In an effort to improve the quality of 3DGS-based inverse rendering under deferred rendering, we propose a novel two-step training approach which (1) exploits mesh extraction and utilizes a hybrid mesh-3DGS representation and (2) applies novel regularization methods to better exploit the mesh. Our experiments show that, under relighting, the proposed method offers significantly better rendering quality than the existing 3DGS-based inverse rendering methods. Compared with the SOTA voxel grid-based inverse rendering method, it gives better rendering quality while offering real-time rendering.

Figures (8)

• Figure 1: (a) Rendering pipeline of the existing relightable 3DGS models, (b) Our proposed deferred rendering.
• Figure 2: Opacity statistics in the TensoIR-Synthetic scenes.
• Figure 3: Example scenes with hidden Gaussian problems.
• Figure 4: Visualization of extracted surface meshes on TensoIR-synthetic scenes.
• Figure 5: Illustration of our proposed scale regularization loss. For a hypothetical mesh triangle, if the largest scale of a bound Gaussian exceeds the scaled circumradius (or target radius, denoted as $r_{sc}$), our regularizer enforces the Gaussian to reduce its scale value.