Light of Normals: Unified Feature Representation for Universal Photometric Stereo

TL;DR

This work tackles universal photometric stereo under unknown lighting by explicitly decoupling illumination from geometry using Light Register Tokens and an Interleaved Attention mechanism, while preserving high-frequency details with a Wavelet Dual-branch Architecture and a Normal-gradient Perception Loss. It introduces PS-Verse, a large-scale synthetic dataset with curriculum learning to support robust training and evaluation. Empirical results demonstrate state-of-the-art normals on DiLiGenT and Luces, improved generalization to real materials, and favorable efficiency compared to prior methods. Collectively, the approach advances practical universal PS for complex lighting and materials, enabling sharper, more reliable 3D reconstructions in unconstrained real-world scenes.

Abstract

Universal photometric stereo (PS) is defined by two factors: it must (i) operate under arbitrary, unknown lighting conditions and (ii) avoid reliance on specific illumination models. Despite progress (e.g., SDM UniPS), two challenges remain. First, current encoders cannot guarantee that illumination and normal information are decoupled. To enforce decoupling, we introduce LINO UniPS with two key components: (i) Light Register Tokens with light alignment supervision to aggregate point, direction, and environment lights; (ii) Interleaved Attention Block featuring global cross-image attention that takes all lighting conditions together so the encoder can factor out lighting while retaining normal-related evidence. Second, high-frequency geometric details are easily lost. We address this with (i) a Wavelet-based Dual-branch Architecture and (ii) a Normal-gradient Perception Loss. These techniques yield a unified feature space in which lighting is explicitly represented by register tokens, while normal details are preserved via wavelet branch. We further introduce PS-Verse, a large-scale synthetic dataset graded by geometric complexity and lighting diversity, and adopt curriculum training from simple to complex scenes. Extensive experiments show new state-of-the-art results on public benchmarks (e.g., DiLiGenT, Luces), stronger generalization to real materials, and improved efficiency; ablations confirm that Light Register Tokens + Interleaved Attention Block drive better feature decoupling, while Wavelet-based Dual-branch Architecture + Normal-gradient Perception Loss recover finer details.

Figures (18)

• Figure 1: (left) Given multi-light images from a fixed viewpoint, LINO UniPS recovers sharper, more faithful normals than UniPS/SDM-UniPS and visually rivals a 3D scanner. (right) On the DiLiGenT, a clear correlation exists between the consistency of encoder features (CSIM/SSIM) and the final reconstruction accuracy (1/MAE).
• Figure 2: Overview of the LINO UniPS architecture. The encoder includes a Wavelet Feature Extractor, a Light Registered Attention Module and a Wavelet Aggregator, which together encode and fuse wavelet and downsample domain features for obtaining unified feature. The decoder is similar to SDM UniPS sdmunips.
• Figure 3: The attention maps for our different Light Register Tokens, derived from the encoder's final-layer feature maps for both real (left) and synthetic (right) data.
• Figure 4: Features from different methods' encoders; rightmost column is variance.
• Figure 5: Results of object inference with masks in Luces and DiLiGenT. Using 16 input images.