ProbeSDF: Light Field Probes for Neural Surface Reconstruction

TL;DR

ProbeSDF introduces a light-field–informed, decoupled radiance model for neural surface reconstruction that separates high-resolution spatial features from low-resolution angular features stored as light-field probes on a coarse grid. A small MLP decodes color from factorized spatial features and SH-based angular embeddings, while SDF geometry is optimized via differentiable rendering in a NeuS-like framework, with strong regularization and a coarse-to-fine training schedule. The approach delivers state-of-the-art or competitive reconstruction quality and enables real-time rendering across diverse datasets (objects and humans) with significant training speedups and memory efficiency. This work offers a practical path to scalable, fast neural surface reconstruction with tunable angular detail and has potential implications for real-time capture, streaming, and interactive 3D content creation.

Abstract

SDF-based differential rendering frameworks have achieved state-of-the-art multiview 3D shape reconstruction. In this work, we re-examine this family of approaches by minimally reformulating its core appearance model in a way that simultaneously yields faster computation and increased performance. To this goal, we exhibit a physically-inspired minimal radiance parametrization decoupling angular and spatial contributions, by encoding them with a small number of features stored in two respective volumetric grids of different resolutions. Requiring as little as four parameters per voxel, and a tiny MLP call inside a single fully fused kernel, our approach allows to enhance performance with both surface and image (PSNR) metrics, while providing a significant training speedup and real-time rendering. We show this performance to be consistently achieved on real data over two widely different and popular application fields, generic object and human subject shape reconstruction, using four representative and challenging datasets.

Figures (24)

• Figure 1: We design a new appearance model for neural surface approaches, which combines high resolution spatial features and lower resolution angular features for improved reconstruction quality, training and inference speed. We plot the chamfer distance as a function of training speed for several baselines on MVMannequins (top) and DTU (bottom).
• Figure 2: Overview of the color prediction for a single voxel inside a $16^3$ tile. The angular features $F_a$ are computed by interpolating and evaluating the spherical harmonics from the 8 corner probes with the reflected vector $\mathbf{r}$. The spatial features $F_s$ are computed as the outer product of three orthogonal planes of resolution $16\times16$, specific to the tile. A small neural network decodes these inputs into a color.
• Figure 3: The reflectivity cannot be encoded in $\mathbf{F}_a$ alone because different incident vectors can map to the same reflected vector (in red). In that case, similar angular features will be obtained for the two viewpoints due to the spatial and angular proximity of the lookups. The angle of incidence disambiguates the two situations.
• Figure 4: Left and center: The lighting direction changes at a lower rate on the surface for distant light sources. Right: Self shadows impose variations in lighting even with directional illumination.
• Figure 5: High-resolution reconstructions for ActorsHQ.