TensoSDF: Roughness-aware Tensorial Representation for Robust Geometry and Material Reconstruction

TL;DR

This work tackles the ill-posed problem of reconstructing geometry and materials from multi-view images for objects with arbitrary reflectance. It introduces TensoSDF, a tensorial SDF representation that, together with roughness-aware fusion of radiance and reflectance fields, yields detailed surfaces and robust material estimates while reducing training time. An explicit mesh–implicit SDF fusion strategy for material estimation further improves relighting quality, achieving state-of-the-art results on synthetic and real data and offering significant speedups. The approach advances practical neural rendering by delivering reliable geometry, high-fidelity materials, and efficient inference for photo-realistic relighting.

Abstract

Reconstructing objects with realistic materials from multi-view images is problematic, since it is highly ill-posed. Although the neural reconstruction approaches have exhibited impressive reconstruction ability, they are designed for objects with specific materials (e.g., diffuse or specular materials). To this end, we propose a novel framework for robust geometry and material reconstruction, where the geometry is expressed with the implicit signed distance field (SDF) encoded by a tensorial representation, namely TensoSDF. At the core of our method is the roughness-aware incorporation of the radiance and reflectance fields, which enables a robust reconstruction of objects with arbitrary reflective materials. Furthermore, the tensorial representation enhances geometry details in the reconstructed surface and reduces the training time. Finally, we estimate the materials using an explicit mesh for efficient intersection computation and an implicit SDF for accurate representation. Consequently, our method can achieve more robust geometry reconstruction, outperform the previous works in terms of relighting quality, and reduce 50% training times and 70% inference time.

Figures (14)

• Figure 1: The network architecture in our geometry reconstruction step. The core of our network is the TensoSDF (on the left), consisting of a tensorial encoder and an MLP decoder, which maps the position of sampled points to SDF values and appearance features. This TensoSDF representation is learned by incorporating the radiance and reflectance fields with the roughness as a balancing weight. Specifically, the joint loss $l_c$ is designed to combine two color losses $l_\mathrm{rad}$ and $l_\mathrm{ref}$, which are computed by rendering the radiance and reflectance fields, respectively. The structure of the reflectance field is shown on the right.
• Figure 2: Illustration of our mesh-SDF fusion strategy. First, we perform ray-mesh intersection to get a rough hit point. Then, we sample $m$ points within a fixed distance inside and outside the surface.
• Figure 3: Comparison of geometry reconstruction among our method, TensoIR jin2023tensoir, NeRO liu2023nero, NeILF++ zhang2023neilf++ and NeuS wang2021neus on our synthetic dataset.
• Figure 4: Comparison of geometry reconstruction among our method, NeILF++ zhang2023neilf++, NeRO liu2023nero and NeuS wang2021neus on the scene from NeRO liu2023nero. There is no available ground-truth mesh in NeRO dataset, and the CD$\downarrow$ loss is computed on the point cloud following NeRO liu2023nero.
• Figure 5: Comparison of geometry reconstruction among our method, NeILF++ zhang2023neilf++, NeRO liu2023nero and NeuS wang2021neus on scenes from TensoIR jin2023tensoir.