Differentiable Display Photometric Stereo

TL;DR

DDPS tackles the challenge of obtaining high-quality surface normals with display photometric stereo by learning display patterns in an end-to-end differentiable framework that couples basis-illumination image formation with an analytic photometric-stereo reconstructor. The approach leverages 3D-printed training data for supervised pattern learning and exploits polarization-based diffuse-specular separation using a polarization camera, enabling more accurate normal estimates on real-world objects. Key contributions include a differentiable image-formation model, a real-data training strategy, and demonstrated robustness to calibration and simplifications, with learned patterns outperforming heuristic designs and existing learning-based photometric stereo methods. The work advances practical 3D reconstruction using off-the-shelf displays and cameras and suggests broader applicability to illumination-camera systems beyond display photometric stereo.

Abstract

Photometric stereo leverages variations in illumination conditions to reconstruct surface normals. Display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we present differentiable display photometric stereo (DDPS), addressing an often overlooked challenge in display photometric stereo: the design of display patterns. Departing from using heuristic display patterns, DDPS learns the display patterns that yield accurate normal reconstruction for a target system in an end-to-end manner. To this end, we propose a differentiable framework that couples basis-illumination image formation with analytic photometric-stereo reconstruction. The differentiable framework facilitates the effective learning of display patterns via auto-differentiation. Also, for training supervision, we propose to use 3D printing for creating a real-world training dataset, enabling accurate reconstruction on the target real-world setup. Finally, we exploit that conventional LCD monitors emit polarized light, which allows for the optical separation of diffuse and specular reflections when combined with a polarization camera, leading to accurate normal reconstruction. Extensive evaluation of DDPS shows improved normal-reconstruction accuracy compared to heuristic patterns and demonstrates compelling properties such as robustness to pattern initialization, calibration errors, and simplifications in image formation and reconstruction.

Figures (10)

• Figure 1: We propose differentiable display photometric stereo, a method that facilitates (a) the learning of display patterns, enabling high-quality reconstruction of surface normals using (b) a monitor and a camera. (c) Capturing a scene with the learned patterns allows for estimating (d) high-quality surface normals.
• Figure 2: Overview of DDPS. DDPS consists of three stages: dataset acquisition, pattern training, and testing.
• Figure 3: Polarimetric imaging system. (a) Imaging system consisting of an LCD monitor and a polarization camera. Decomposed (b) diffuse image and specular image using linearly-polarized light emitted from the monitor.
• Figure 4: Training dataset creation with 3D printing. To learn display patterns, we propose to use (a) 3D-printed objects that have corresponding (b) known ground-truth 3D models. (c) We extract the silhouette $S$ from the averaged basis images and (d) align the ground-truth 3D models with the captured image as depicted with the fitted silhouette in red on top of the average image. (e) We obtain a ground-truth normal map from the fitted 3D model.
• Figure 5: Differentiable framework. Using 3D-printed objects as a dataset allows for simulating real-world captured images with a differentiable image formation. We reconstruct high-fidelity surface normals using differentiable photometric stereo from the simulated captured images.