Physics-Free Spectrally Multiplexed Photometric Stereo under Unknown Spectral Composition

TL;DR

A unique, physics-free network architecture is introduced, SpectraM-PS, that effectively processes multiplexed images to determine surface normals across a wide range of conditions and material types, without relying on specific physically-based knowledge.

Abstract

In this paper, we present a groundbreaking spectrally multiplexed photometric stereo approach for recovering surface normals of dynamic surfaces without the need for calibrated lighting or sensors, a notable advancement in the field traditionally hindered by stringent prerequisites and spectral ambiguity. By embracing spectral ambiguity as an advantage, our technique enables the generation of training data without specialized multispectral rendering frameworks. We introduce a unique, physics-free network architecture, SpectraM-PS, that effectively processes multiplexed images to determine surface normals across a wide range of conditions and material types, without relying on specific physically-based knowledge. Additionally, we establish the first benchmark dataset, SpectraM14, for spectrally multiplexed photometric stereo, facilitating comprehensive evaluations against existing calibrated methods. Our contributions significantly enhance the capabilities for dynamic surface recovery, particularly in uncalibrated setups, marking a pivotal step forward in the application of photometric stereo across various domains.

Figures (33)

• Figure 1: (Left) Illustration of our SpectraM-PS. Our method recovers a surface normal map from a spectrally multiplexed image. The spectral/spatial composition for generating the observations is unknown. There is potential for a mismatch between the sensor's spectral sensitivity and the light source's spectral distribution, which may lead to crosstalk. (Right) By applying our method to individual frames of a video, the normal map of dynamic surfaces can be recovered.
• Figure 2: SpectraM-PS involves decomposing a spectrally multiplexed image into independent channels. The Global Feature Encoder extracts a feature map from each channel. The surface vector is then recovered by the Dual-scale Surface Normal Decoder at each pixel. We adopt a dual-scale approach to preserve the entire shape, while employing patch-embedding techniques to enhance local surface details.
• Figure 3: (Left) Comparison of SpectraM-PS and SDM-UniPS Ikehata2023 on six temporally multiplexed PS images. Due to the patch-wise basis of SpectraM-PS, fine details are better recovered. (Right) Illustration of different lighting conditions in PS-Multiplex.
• Figure 4: Objects in SpectraM14.
• Figure 6: Evaluation on SpectraM14. Full results are available in the supplementary.