PRM: Photometric Stereo based Large Reconstruction Model

TL;DR

PRM tackles the challenge of reconstructing high-fidelity 3D meshes under complex appearances by integrating photometric stereo with a large reconstruction model. It employs a real-time split-sum physically-based rendering pipeline and an explicit mesh representation to provide rich photometric cues and differentiable supervision, enabling robust geometry under glossy and varying materials. The two-stage optimization leverages both offline and mesh-based representations, with losses that jointly supervise color, albedo, lighting maps, normals, and depth. Across GSO and Omni3D, PRM substantially surpasses prior methods in 3D geometry and 2D rendering quality, demonstrating strong resilience to lighting and material variations and enabling downstream tasks like relighting and material editing.

Abstract

We propose PRM, a novel photometric stereo based large reconstruction model to reconstruct high-quality meshes with fine-grained local details. Unlike previous large reconstruction models that prepare images under fixed and simple lighting as both input and supervision, PRM renders photometric stereo images by varying materials and lighting for the purposes, which not only improves the precise local details by providing rich photometric cues but also increases the model robustness to variations in the appearance of input images. To offer enhanced flexibility of images rendering, we incorporate a real-time physically-based rendering (PBR) method and mesh rasterization for online images rendering. Moreover, in employing an explicit mesh as our 3D representation, PRM ensures the application of differentiable PBR, which supports the utilization of multiple photometric supervisions and better models the specular color for high-quality geometry optimization. Our PRM leverages photometric stereo images to achieve high-quality reconstructions with fine-grained local details, even amidst sophisticated image appearances. Extensive experiments demonstrate that PRM significantly outperforms other models.

Figures (19)

• Figure 1: Top left: PRM is capable of reconstructing high-quality meshes with fine-grained local details even under complex image appearances, such as specular highlights and dark appearances. Right: We demonstrate a scene comprising diverse 3D objects generated by our models. Bottom left: A zoomed-in visualization of the scene highlights these details more clearly.
• Figure 2: Overview of our framework. During training, photometric stereo images are rendered using PBR with randomly varied materials, lighting, and camera poses, along with depth, normal, albedo, and lighting maps. Images are encoded as a mesh through the network. All associated maps, along with the images, are used for supervision. During inference, an optional multi-view diffusion model takes a single image as input and outputs multi-view images, which are then fed into the network for mesh prediction. Relighting and material editing functionalities are also supported.
• Figure 3: Comparison on shiny objects.
• Figure 4: Qualitative comparisons with state-of-the art methods and ground truth for single-view reconstruction task. PRM reconstructs the highest quality 3D mesh and provides a more accurate texture prediction from input photographs compared to the others.
• Figure 5: Ablation study to validate the effectiveness of each individual component.