Hardware-aware Coding Function Design for Compressive Single-Photon 3D Cameras

TL;DR

The paper tackles the challenge of data-rate limits in compressive single-photon ToF cameras by introducing hardware-aware constrained optimization to jointly design the illumination waveform and the coding matrix. By formulating a differentiable optimization with a Gaussian impulse response and peak-power constraints, the authors show that optimized coding functions outperform traditional compressive histograms across bandwidth and power regimes, and adapt to non-ideal impulse responses. Key contributions include a gradient-descent framework for f(t) and D, a differentiable ZNCC decoder, and demonstrations of memory-efficient code representations suitable for on-chip implementation. The results suggest practical pathways to deploy low-power, high-resolution ToF imaging with SPAD sensors in real-world hardware-limited systems.

Abstract

Single-photon cameras are becoming increasingly popular in time-of-flight 3D imaging because they can time-tag individual photons with extreme resolution. However, their performance is susceptible to hardware limitations, such as system bandwidth, maximum laser power, sensor data rates, and in-sensor memory and compute resources. Compressive histograms were recently introduced as a solution to the challenge of data rates through an online in-sensor compression of photon timestamp data. Although compressive histograms work within limited in-sensor memory and computational resources, they underperform when subjected to real-world illumination hardware constraints. To address this, we present a constrained optimization approach for designing practical coding functions for compressive single-photon 3D imaging. Using gradient descent, we jointly optimize an illumination and coding matrix (i.e., the coding functions) that adheres to hardware constraints. We show through extensive simulations that our coding functions consistently outperform traditional coding designs under both bandwidth and peak power constraints. This advantage is particularly pronounced in systems constrained by peak power. Finally, we show that our approach adapts to arbitrary parameterized impulse responses by evaluating it on a real-world system with a non-ideal impulse response function.

Figures (11)

• Figure 1: Hardware-aware coding function design for compressive single-photon 3D cameras. (a) Overview of a single-photon 3D camera based on the time-of-flight principle, which illuminates a scene with a pulsed laser and detects the reflected signal at a single-photon detector. (b) The ideal source illumination will have infinite bandwidth and power, but practical hardware considerations demonstrate illumination bandwidth limitations, peak power limitations, and non-ideal impulse response functions. (c) Simulated depth map and depth error results using full-res. histogramming, compressive histogramming sheehan2021sketching, and our optimized coding functions with finite bandwidth and finite peak power.
• Figure 2: Overview of hardware-aware optimized coding functions. Conventional compressive histograms (left) perform optimally with the ideal pulsed illumination. When hardware constraints, such as bandwidth (middle) and peak power (right), are introduced, compressive histograms become suboptimal, and the coding matrix and source illumination must be adjusted to prevent performance degradation. To address this, we propose a method to optimize both the illumination and coding matrix (or coding functions) based on bandwidth and/or peak power constraints. By adapting the coding functions, we demonstrate improved performance compared to compressive histograms.
• Figure 3: Gray codes subjected to strict bandwidth constraints. Gray-based gutierrez2022compressive compressive histograms cannot efficiently encode the illumination's waveform when severely band-limited.
• Figure 4: Simulated results with finite bandwidth and infinite peak power. This figure shows the mean absolute error on the left and the root mean squared error on the right, obtained from Monte Carlo simulations using the different coding schemes, evaluated at an IRF width of $\sigma = 30\Delta$ and infinite peak power. The optimized coding functions achieve the closest accuracy to the FRH, regardless of SBR, when the mean photon count is above $150$. For photon counts below $150$, both the Gray and the optimized codings exhibit a large gap between RMSE and MAE due to the presence of outliers.
• Figure 5: Hardware-constrained optimized coding functions. This figure plots the optimized coding functions found for different bandwidths and peak power constraints. The columns indicate the width of the IRF $h(t)$ and the rows indicate the peak factor $p^{\text{factor}}$. For each $p^{\text{factor}}$ and $\sigma$ we display the coding matrix $D$ on the top right, display the $2$nd and $7$th row of $D$ on the top left, and display the pulsed illumination (light blue) and the optimized illumination (blue) incident on the sensor at the bottom. Here we optimize our codes with a constant mean photon count of $\Phi^{\mathrm{sig}}=1000$. Note that the middle of the illumination plot is zero.