OMG: Opacity Matters in Material Modeling with Gaussian Splatting

TL;DR

This work augments the opacity term by introducing a neural network that takes as input material properties to provide modeling of cross section and a physically correct activation function, and achieves significant improvements universally in terms of novel view synthesis and material modeling.

Abstract

Decomposing geometry, materials and lighting from a set of images, namely inverse rendering, has been a long-standing problem in computer vision and graphics. Recent advances in neural rendering enable photo-realistic and plausible inverse rendering results. The emergence of 3D Gaussian Splatting has boosted it to the next level by showing real-time rendering potentials. An intuitive finding is that the models used for inverse rendering do not take into account the dependency of opacity w.r.t. material properties, namely cross section, as suggested by optics. Therefore, we develop a novel approach that adds this dependency to the modeling itself. Inspired by radiative transfer, we augment the opacity term by introducing a neural network that takes as input material properties to provide modeling of cross section and a physically correct activation function. The gradients for material properties are therefore not only from color but also from opacity, facilitating a constraint for their optimization. Therefore, the proposed method incorporates more accurate physical properties compared to previous works. We implement our method into 3 different baselines that use Gaussian Splatting for inverse rendering and achieve significant improvements universally in terms of novel view synthesis and material modeling.

Figures (12)

• Figure 1: An illustration of the motivation that opacity depends on materials. Consider two translucent materials gas and glass. Left: a glass body that is hit by a light lets the light pass through with little reduction in intensity. Right: a gas that is hit by the same light make the light extinct inside it by absorbing it completely. This comparisons motivate our work. We perceive each Gaussian blob, paired with material properties, as an absorbing body that has its own absorption coefficient represented by opacity, which should consider material properties as influencing factors.
• Figure 2: Pipeline overview. We add an additional block to 3DGS-based inverse rendering methods liang2024gskerbl20233djiang2024gaussianshadergao2023relightable. Specifically, instead of modeling opacity as a standalone parameter as done by previous works, we augment it against material. By introducing a neural network that takes material properties as input and output cross section and multiplying the opacity with it, we are able to incorporate the Bouguer-Beer-Lambert law into the model. During optimization, material properties not only receive the gradients from color through differentiable PBR, but also the gradients from alpha enforced by the neural network. By doing so, we add an additional constraint to material properties that makes the overall pipeline strictly follow the Bouguer-Beer-Lambert law.
• Figure 3: An illustration of the Bouguer-Beer-Lambert law. Consider a ray passing through an absorbing body consists of particles of some type. The cross section depends on the area that each particle would affect (usually bigger than the size of the particle and equals the size of the particle if treated as a hard sphere). When traveling, the intensity of the light would be reduced by the areas that each particle affected. The reduction in intensity is therefore affected by the area around the particles and the number of particles, corresponding to cross section $\sigma$ and number density $n$ in the main paper.
• Figure 4: Qualitative comparison of albedo, relighting and roughness on Synthetic4Relight dataset zhang2022modeling. Our method is able to decouple specular effects from albedo compared to R3DG, the baseline we implemented our method on. It is also capable of providing roughness estimation more precisely.We add a gamma correction to the relighting results to make the visual differences clearer.
• Figure 5: Qualitative comparison on Shiny Blender verbin2022ref and Glossy Synthetic liu2023nero datasets. Our method is able to provide satisfactory normal estimation compared to the baseline. Notice that our method does not include any additional supervision or constraints for normal estimation. Our method is also able to provide much more accurate view synthesis results when doing physically-based rendering. These results suggest the significance of introducing the physically correct model for inverse rendering.