Radiance Fields from Photons

TL;DR

This work introduces quanta radiance fields, a novel class of neural radiance fields that are trained at the granularity of individual photons using single-photon cameras (SPCs), and demonstrates high-fidelity reconstructions under high-speed motion, in low light, and for extreme dynamic range settings.

Abstract

Neural radiance fields, or NeRFs, have become the de facto approach for high-quality view synthesis from a collection of images captured from multiple viewpoints. However, many issues remain when capturing images in-the-wild under challenging conditions, such as low light, high dynamic range, or rapid motion leading to smeared reconstructions with noticeable artifacts. In this work, we introduce quanta radiance fields, a novel class of neural radiance fields that are trained at the granularity of individual photons using single-photon cameras (SPCs). We develop theory and practical computational techniques for building radiance fields and estimating dense camera poses from unconventional, stochastic, and high-speed binary frame sequences captured by SPCs. We demonstrate, both via simulations and a SPC hardware prototype, high-fidelity reconstructions under high-speed motion, in low light, and for extreme dynamic range settings.

Figures (13)

• Figure 1: Reconstruction under challenging scenario: We simulate a high-speed (${\hbox{$\sim$}} 72$km/h) drone fly-through of an indoor scene which has $16$ stops of dynamic range. We train a conventional motion-aware NeRF with tonemapped images lee2023exblurf, one with raw linear intensity conventional images like in rawnerf, and a QRF with simulated binary frames from this scene. In ambient light, the ExBluRF and RawNeRF reconstructions suffer from various artifacts, while the QRF model captures the full dynamic range of the scene with no noticeable blur. In the extreme case of lowering the light levels by $100\times$, the QRF reconstruction remains recognizable, while baselines fail.
• Figure 1: Hardware Setup
• Figure 2: View Diversity and Extrapolation: Radiance fields trained using single photon data perform better view extrapolation (novel views that are significantly different than the training poses) and degrade more gracefully than ones trained with conventional frames, given the same total capture time. The training viewpoints in both cases span the same trajectory, that is, both datasets have the same pose diversity, however, the denser sampling provided by the single photon camera better constrains the scene's geometry leading to significantly improved reconstructions and better generalization.
• Figure 2: Final pose deviations vs. initial
• Figure 3: Camera Pose Optimization: The trajectory of the camera is co-optimized with the radiance field. Due to the large number of camera poses to optimize and the noisy binary measurements, a strong smoothing regularizer on the poses is needed. Without it, poses settle in noisy local minima, which affects the final reconstruction quality.