ROSA: Reconstructing Object Shape and Appearance Textures by Adaptive Detail Transfer

TL;DR

ROSA addresses the ill-posed problem of reconstructing an object's shape and SVBRDF from limited collocated-light images by combining adaptive mesh refinement guided by texture normals and curvature with a tile-based, decoder-generated high-resolution appearance texture. A novel normal loss transfers fine geometric details from the texture normals to the mesh, while curvature-driven controls govern local refinement and smoothing to keep the representation compact. Appearance textures are generated via a tile-based approach using a pre-trained autoencoder, enabling arbitrary-resolution textures without enlarging the base mesh. Quantitative and qualitative evaluations on synthetic and real data show ROSA delivers high-fidelity reconstructions with more compact meshes and realistic highlights compared to prior methods, facilitating accurate relighting and rendering.

Abstract

Reconstructing an object's shape and appearance in terms of a mesh textured by a spatially-varying bidirectional reflectance distribution function (SVBRDF) from a limited set of images captured under collocated light is an ill-posed problem. Previous state-of-the-art approaches either aim to reconstruct the appearance directly on the geometry or additionally use texture normals as part of the appearance features. However, this requires detailed but inefficiently large meshes, that would have to be simplified in a post-processing step, or suffers from well-known limitations of normal maps such as missing shadows or incorrect silhouettes. Another limiting factor is the fixed and typically low resolution of the texture estimation resulting in loss of important surface details. To overcome these problems, we present ROSA, an inverse rendering method that directly optimizes mesh geometry with spatially adaptive mesh resolution solely based on the image data. In particular, we refine the mesh and locally condition the surface smoothness based on the estimated normal texture and mesh curvature. In addition, we enable the reconstruction of fine appearance details in high-resolution textures through a pioneering tile-based method that operates on a single pre-trained decoder network but is not limited by the network output resolution.

Figures (10)

• Figure 1: Exemplary renderings in Unity emphasizing artifacts from geometric surface details being approximated in normal textures instead of faithfully representing them in the mesh geometry.
• Figure 2: Overview of our inverse rendering framework, which performs decoder-based texture estimation and triangle mesh optimization with non-uniform smoothing. The regional increase of the mesh resolution and its smoothness during optimization are controlled by the estimated texture normal and the current mesh curvature.
• Figure 3: Architecture of our decoder for texture estimation which is pre-trained as part of an autoencoder and later finetuned.
• Figure 4: Geometry reconstruction over time. While in the early phase surface details are represented in the texture normals, these details are progressively transferred to the geometry in the later phase when the overall geometry is converging to the true shape.
• Figure 5: Damping functions used to increase the stability of the reconstruction and to transfer the geometry information from the texture normal to the mesh surface.