Deep Learning Methods for Calibrated Photometric Stereo and Beyond

TL;DR

The paper surveys deep learning approaches to calibrated photometric stereo under non-Lambertian reflectance, presenting a taxonomy by input processing, supervision, and network architecture. It highlights per-pixel, all-pixel, and hybrid methods, discussing their strengths, limitations, and strategies to address challenges such as sparse inputs, spatially varying BRDF, and blurry details. It reviews data sets (Blobby/Sculpture, CyclePS) and real synthetic benchmarks (DiLiGenT and derivatives), showing that DL-based PS often surpasses traditional methods in accuracy, especially on complex materials. Finally, it outlines future directions, including universal lighting models (UniPS, SDM-UniPS), improved fusion via self-attention, and neural rendering integrations (NeRF-based PS) to enable robust, scalable surface normal estimation in real-world scenarios.

Abstract

Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues, i.e., modeling the relationship between surface orientation and intensity at each pixel. Photometric stereo prevails in superior per-pixel resolution and fine reconstruction details. However, it is a complicated problem because of the non-linear relationship caused by non-Lambertian surface reflectance. Recently, various deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces. This paper provides a comprehensive review of existing deep learning-based calibrated photometric stereo methods. We first analyze these methods from different perspectives, including input processing, supervision, and network architecture. We summarize the performance of deep learning photometric stereo models on the most widely-used benchmark data set. This demonstrates the advanced performance of deep learning-based photometric stereo methods. Finally, we give suggestions and propose future research trends based on the limitations of existing models.

Figures (11)

• Figure 1: The schematic of photometric stereo. The orange box shows the general surface reflectance.
• Figure 2: Overview of the main deep learning methods for calibrated photometric stereo.
• Figure 3: The illustration of the observation maps ikehata2018cnn. Here, a, b, and c represent the number of input images (lights), while 1, 2, and 3 denote the index of pixel positions.
• Figure 4: Per-pixel methods for sparse input images. SPLINE-Net zheng2019spline uses the lighting interpolation network to generate dense observation maps, while LMPS li2019learning applies the connection table used to select the most relevant illuminant directions in the sparse observation maps.
• Figure 5: Examples of the predictions and error maps on spatially varying BRDF, from the object "Harvest" in the DiLiGenT data set shi2019benchmark. NA-PSN is short for NormAttention-PSN. The numbers reveal the mean angular error in degrees.