Spin-UP: Spin Light for Natural Light Uncalibrated Photometric Stereo

TL;DR

The paper tackles Natural Light Uncalibrated Photometric Stereo (NaUPS) by introducing Spin-UP, an unsupervised method that jointly recovers surface normals, environment light, and isotropic reflectance under general natural illumination. It relies on a novel Spin Light Setup that rotates the object and camera to tie observations to a low-DOF light model, plus a boundary-based light prior to provide a reliable illumination initialization. A neural inverse rendering framework models geometry with a neural depth field, spatially varying isotropic reflectance, and a light representation built from spherical Gaussian bases, optimized with inverse rendering losses; two training strategies—Interval Sampling and Shrinking Range Computing—accelerate convergence. Experiments on synthetic and real datasets show Spin-UP achieving state-of-the-art or competitive performance across shapes, lights, and materials, with ablations validating the importance of the light initialization, filtering, and training strategies for both accuracy and efficiency.

Abstract

Natural Light Uncalibrated Photometric Stereo (NaUPS) relieves the strict environment and light assumptions in classical Uncalibrated Photometric Stereo (UPS) methods. However, due to the intrinsic ill-posedness and high-dimensional ambiguities, addressing NaUPS is still an open question. Existing works impose strong assumptions on the environment lights and objects' material, restricting the effectiveness in more general scenarios. Alternatively, some methods leverage supervised learning with intricate models while lacking interpretability, resulting in a biased estimation. In this work, we proposed Spin Light Uncalibrated Photometric Stereo (Spin-UP), an unsupervised method to tackle NaUPS in various environment lights and objects. The proposed method uses a novel setup that captures the object's images on a rotatable platform, which mitigates NaUPS's ill-posedness by reducing unknowns and provides reliable priors to alleviate NaUPS's ambiguities. Leveraging neural inverse rendering and the proposed training strategies, Spin-UP recovers surface normals, environment light, and isotropic reflectance under complex natural light with low computational cost. Experiments have shown that Spin-UP outperforms other supervised / unsupervised NaUPS methods and achieves state-of-the-art performance on synthetic and real-world datasets. Codes and data are available at https://github.com/LMozart/CVPR2024-SpinUP.

Figures (19)

• Figure 1: The proposed image capturing setup. Left: an illustration of image-capturing equipment consisting of a rotatable platform, a camera, and the target object. We spin the platform in $360^\circ$ and capture images of the object. The object and camera rotate together with the platform. Right-top: Four observed images at different positions. Right-bottom: Ground truth environment light. Dashed color boxes indicate the corresponding camera views.
• Figure 2: The proposed light initialization method in Spin-UP. We crop the boundary pixels $m_b$ and normal $\boldsymbol{n}_b$ from input images. Then, we remap them on the sphere and rotate them with their corresponding rotations $\boldsymbol{R}$. Based on a light probe composed of gray-scale boundary pixels, we optimize the SG light model to obtain the environment light.
• Figure 3: An illustration of mismatched light source positions. Rows from top to bottom: an illustration of reflection on materials with different roughness; the initial environment light before applying any filters given objects' boundary pixels; the ground truth environment light. Objects are (a) the diffuse-dominant sphere, (b) the diffuse and specular mixture sphere, and (c) the specular dominant sphere. The yellow and red lines in rows 2 and 3 indicate a rough position of the light sources.
• Figure 4: Proposed training strategies. (Top) Interval sampling (IS). The high-resolution images are down-sampled into several low-resolution sub-images by extracting pixels with an interval of $N_B$, where $N_B=2$ in this example. (Bottom) Shrinking range computing (SRC). Far points (yellow circles) that are $k$ ($k=3$ in this example) point away w.r.t the query position is selected to interpolate the close points (blue circles)'s depth for normal calculation. During optimization, $k$ will be gradually reduced to 1.
• Figure 5: The visual quality comparison of light between the estimated one by Spin-UP (columns 2-5) and the ground truth (column 1) on the four groups, i.e, shape (row 1), reflectance (row 2), spatially varying material (row 3), and light group (row 4-5). The intensity of the estimated light maps is scaled for visualization.