Single-Photon 3D Imaging with Equi-Depth Photon Histograms

TL;DR

This work tackles bandwidth and memory bottlenecks in SPAD-based 3D imaging by replacing conventional equi-width histograms with equi-depth histograms (EDH) and online, count-free bin-boundary estimation. The proposed PEDH pipeline compresses photon data while preserving peak information, and couples it with a learning-based DeePEDH distance estimator to produce high-quality depth maps under challenging lighting. The approach yields substantial reductions in data bandwidth and in-pixel memory requirements, while enabling robust downstream tasks such as RGBD visual odometry, dense 3D reconstruction, and RGBD semantic segmentation on resource-constrained platforms. This enables practical, high-resolution 3D perception for mobile devices and AR/VR sensing, with demonstrated gains over EW-based SPCs in both simulation and hardware-emulation scenarios.

Abstract

Single-photon cameras present a promising avenue for high-resolution 3D imaging. They have ultra-high sensitivity -- down to individual photons -- and can record photon arrival times with extremely high (sub-nanosecond) resolution. Single-photon 3D cameras estimate the round-trip time of a laser pulse by forming equi-width (EW) histograms of detected photon timestamps. Acquiring and transferring such EW histograms requires high bandwidth and in-pixel memory, making SPCs less attractive in resource-constrained settings such as mobile devices and AR/VR headsets. In this work we propose a 3D sensing technique based on equi-depth (ED) histograms. ED histograms compress timestamp data more efficiently than EW histograms, reducing the bandwidth requirement. Moreover, to reduce the in-pixel memory requirement, we propose a lightweight algorithm to estimate ED histograms in an online fashion without explicitly storing the photon timestamps. This algorithm is amenable to future in-pixel implementations. We propose algorithms that process ED histograms to perform 3D computer-vision tasks of estimating scene distance maps and performing visual odometry under challenging conditions such as high ambient light. Our work paves the way towards lower bandwidth and reduced in-pixel memory requirements for SPCs, making them attractive for resource-constrained 3D vision applications. Project page: $\href{https://www.computational.camera/pedh}{https://www.computational.camera/pedh}$

Figures (11)

• Figure 1: Our proposed DeePEDH pipeline enables SPCs to be used in resource-constrained applications. (a--c) Conventional 3D sensing pipeline constructs equi-width (EW) histograms that require high in-pixel memory and cause a data bottleneck. (d) Existing methods resort to low resolution EW histograms at the cost of poor distance resolution. (e--g) DeePEDH uses more efficient on-sensor compression scheme through equi-depth (ED) histograms combined with a deep neural network distance map estimator. (h) DeePEDH provides accurate high resolution distance maps with $10-100\times$ lower bandwidth. (i--k) Various downstream vision tasks can benefit from high quality distance maps generated using DeePEDH.
• Figure 1: Choosing the best scaling factor $K$: (a) Plots illustrate the effect of $K$ on different ($\Phi_\text{sig}, \Phi_\text{bkg}$) pairs for a wide range of distance values. (b) Plots illustrate the average errors over eight different ($\Phi_\text{sig}, \Phi_\text{bkg}$) pairs and varying scene distances. The average error increases as $K$ increases.
• Figure 2: Equi-width vs. equi-depth histograms for peaky data. (a) Most bins of an 8-bin EW histogram are wasted on background photons; a single bin B2 gives a coarse estimate of the true peak. (b) An ED histogram captures this "peaky" transient distribution better with a cluster of narrower bins around the true peak.
• Figure 2: Choosing the best decay parameter $\gamma$: (a) Plots illustrate the effect of $\gamma$ on different ($\Phi_\text{sig}, \Phi_\text{bkg}$) pairs for a range of distance values. (b) Plots illustrate the average errors over eight different ($\Phi_\text{sig}, \Phi_\text{bkg}$) pairs and varying scene distances. The average errors are least for $\gamma = 0.999$.
• Figure 3: A binner element used for tracking ED histogram boundaries. (a) A median-finding binner splits incident photons into early ($E_n$) and late photons ($L_n$) compared to the current CV ($C_n$). As the CV is greater than the true median, $E_n/(E_n + L_n) > 1/2$, hence the step size $S_n$ is negative. (b) Our optimized stepping strategy uses a sequence of step sizes to achieve faster convergence and lower variance compared to earlier fixed-stepping binner design racingspad.