GlossyGS: Inverse Rendering of Glossy Objects with 3D Gaussian Splatting

TL;DR

GloucesterGS, an innovative 3D-GS-based inverse rendering framework that aims to precisely reconstruct the geometry and materials of glossy objects by integrating material priors, is introduced with the use of micro-facet geometry segmentation prior to reduce the intrinsic ambiguities and improve the decomposition of geometries and materials.

Abstract

Reconstructing objects from posed images is a crucial and complex task in computer graphics and computer vision. While NeRF-based neural reconstruction methods have exhibited impressive reconstruction ability, they tend to be time-comsuming. Recent strategies have adopted 3D Gaussian Splatting (3D-GS) for inverse rendering, which have led to quick and effective outcomes. However, these techniques generally have difficulty in producing believable geometries and materials for glossy objects, a challenge that stems from the inherent ambiguities of inverse rendering. To address this, we introduce GlossyGS, an innovative 3D-GS-based inverse rendering framework that aims to precisely reconstruct the geometry and materials of glossy objects by integrating material priors. The key idea is the use of micro-facet geometry segmentation prior, which helps to reduce the intrinsic ambiguities and improve the decomposition of geometries and materials. Additionally, we introduce a normal map prefiltering strategy to more accurately simulate the normal distribution of reflective surfaces. These strategies are integrated into a hybrid geometry and material representation that employs both explicit and implicit methods to depict glossy objects. We demonstrate through quantitative analysis and qualitative visualization that the proposed method is effective to reconstruct high-fidelity geometries and materials of glossy objects, and performs favorably against state-of-the-arts.

Figures (16)

• Figure 1: We present an innovative 3D-GS-based inverse rendering framework that aims to precisely reconstruct the geometries and materials of glossy objects by integrating priors from glossy appearance analysis. The well disentangled geometric and material properties allow us to achieve high-quality relighting (the second and third columns) and material editing (the fourth and fifth columns), especially on highly glossy objects.
• Figure 2: Illustration of the inverse rendering pipeline for our GlossyGS. The initial points are used to generate anchors, which carry the macroscopic features. These features are fed to GS decoder and material encoder, generating neural 3D Gaussians and neural materials (BRDF). Normal map prefiltering is used as a prefilter for these neural Gaussians and materials to obtain the corresponding material maps. The final rendering is shading of these maps and differentiable environment lighting, supervised by ground-truth images and micro-facet geometry segmentation prior.
• Figure 3: Normal Map Prefiltering.$\mathbf{n_1}, \mathbf{n_2}, \mathbf{N}$ represent the normal of Gaussian 1, Gaussian 2, and the true surface, respectively. $c_1, c_2, c_s$ represent the shading results of Gaussian 1, Gaussian 2, and the true surface, respectively. $\mathbf{C}$ represents the color of the pixel, and $w_1$ and $w_2$ represent the weights for $\alpha$-blending. (a) is the current approach used by 3D-GS-based IR methods gao2023relightablejiang2023gaussianshader. (b) On the other hand, our normal map prefiltering considers the nonlinearity of normals with respect to specular color (c) which has a significant impact on the rendering results of glossy objects.
• Figure 4: Ambiguities Between Macroscopic Normal and Microscopic Roughness.$\boldsymbol{\omega_o},\mathbf{n},\mathbf{t},r$ represent outgoing view direction, normal, reflected light, and roughness respectively. (a) represents large roughness and simple normal, while (b) is the opposite, with complex normal and each plane can be considered as a specular reflection with smaller roughness. Both can represent the same reflective object, leading to the occurrence of Case 1 and Case 2 in (c).
• Figure 5: Visualizing the results of micro-facet geometry segmentation and the effect of the segmentation prior. Without the segmentation prior, the reconstructed roughness is noisy and lacks smoothness. Note that there is no available ground-truth roughness in these examples.