DiffTrans: Differentiable Geometry-Materials Decomposition for Reconstructing Transparent Objects

TL;DR

This work proposes a differentiable rendering framework for transparent objects, dubbed DiffTrans, which allows for efficient decomposition and reconstruction of the geometry and materials of transparent objects, thereby reconstructing transparent objects accurately in intricate scenes with diverse topology and complex texture.

Abstract

Reconstructing transparent objects from a set of multi-view images is a challenging task due to the complicated nature and indeterminate behavior of light propagation. Typical methods are primarily tailored to specific scenarios, such as objects following a uniform topology, exhibiting ideal transparency and surface specular reflections, or with only surface materials, which substantially constrains their practical applicability in real-world settings. In this work, we propose a differentiable rendering framework for transparent objects, dubbed DiffTrans, which allows for efficient decomposition and reconstruction of the geometry and materials of transparent objects, thereby reconstructing transparent objects accurately in intricate scenes with diverse topology and complex texture. Specifically, we first utilize FlexiCubes with dilation and smoothness regularization as the iso-surface representation to reconstruct an initial geometry efficiently from the multi-view object silhouette. Meanwhile, we employ the environment light radiance field to recover the environment of the scene. Then we devise a recursive differentiable ray tracer to further optimize the geometry, index of refraction and absorption rate simultaneously in a unified and end-to-end manner, leading to high-quality reconstruction of transparent objects in intricate scenes. A prominent advantage of the designed ray tracer is that it can be implemented in CUDA, enabling a significantly reduced computational cost. Extensive experiments on multiple benchmarks demonstrate the superior reconstruction performance of our DiffTrans compared with other methods, especially in intricate scenes involving transparent objects with diverse topology and complex texture. The code is available at https://github.com/lcp29/DiffTrans.

Figures (18)

• Figure 1: Overview of the proposed DiffTrans. Our approach reconstructs the geometry and materials of the transparent objects with diverse topology and complex texture from a set of multi-view images using differentiable mesh ray tracer, enabling scene editing capabilities such as relighting.
• Figure 2: The framework of our DiffTrans. We commence by reconstructing the initial geometry from multi-view mask images, and recovering the environment light radiance field by employing pixels out of the mask regions (\ref{['sec:initial']}). In the refine phase, we first define the light decay within absorptive medium, and the Fresnel term used for blending in-object radiance and out-of-object radiance (\ref{['sec:light']}). Then we optimize the geometry, IoR and absorption rate of the transparent objects simultaneously via our differentiable recursive mesh ray tracer, with geometry and material regularization (\ref{['sec:raytracer']}).
• Figure 3: The recursive rendering process demonstrated with a light path. $L_{iR}'$ and $L_{iR}$ are the radiance of the reflected ray, which is abbreviated as a single variable for simplicity. $T_i$, $R_i$, $T_i'$ and $R_i'$ are calculated with \ref{['eq:fresnelr']} and \ref{['eq:fresnelt']}.
• Figure 4: Qualitative comparisons of the reconstructed geometry on bunny, horse an monkey scenes.
• Figure 5: Qualitative comparison of relighting results on the horse, monkey, and flower scenes.