Joint Depth and Reflectivity Estimation using Single-Photon LiDAR

TL;DR

This work tackles the challenge of jointly estimating depth and reflectivity in fast-moving SP-LiDAR scenes, where timestamps encode both quantities. It provides a theoretical foundation showing mutual information sharing between depth and reflectivity under nonzero background, and introduces SPLiDER, a two-channel neural network with cross-modal information fusion (CCAM), optical-flow feature alignment, and progressive multi-scale reconstruction. The method leverages timestamp frames directly, enabling robust joint reconstruction where traditional 3D histograms and single-modality methods fail, particularly at low photon counts and high motion. Across synthetic and real data, SPLiDER achieves superior depth and reflectivity accuracy, with ablations confirming the critical role of feature sharing and flow alignment for improved performance.

Abstract

Single-Photon Light Detection and Ranging (SP-LiDAR is emerging as a leading technology for long-range, high-precision 3D vision tasks. In SP-LiDAR, timestamps encode two complementary pieces of information: pulse travel time (depth) and the number of photons reflected by the object (reflectivity). Existing SP-LiDAR reconstruction methods typically recover depth and reflectivity separately or sequentially use one modality to estimate the other. Moreover, the conventional 3D histogram construction is effective mainly for slow-moving or stationary scenes. In dynamic scenes, however, it is more efficient and effective to directly process the timestamps. In this paper, we introduce an estimation method to simultaneously recover both depth and reflectivity in fast-moving scenes. We offer two contributions: (1) A theoretical analysis demonstrating the mutual correlation between depth and reflectivity and the conditions under which joint estimation becomes beneficial. (2) A novel reconstruction method, "SPLiDER", which exploits the shared information to enhance signal recovery. On both synthetic and real SP-LiDAR data, our method outperforms existing approaches, achieving superior joint reconstruction quality.

Figures (22)

• Figure 1: Different Processing Methods of Timestamp Data and Corresponding Results: (a) The SPAD sensor array captures noisy timestamp data from a dynamic scene at a high speed. (b) To mitigate noise in raw data, a conventional approach involves clustering multiple detections to form a $3$D cube. Subsequently, the object's reflectivity and depth are determined by identifying the height and the location of the peak through algorithms such as maximum likelihood estimation. (c) We propose SPLiDER, a deep learning framework that leverages individual timestamp frames. (d) Conventional algorithms suffer from blurry results due to a long integration time. (e) Our proposed method yields better results.
• Figure 2: Performance Analysis of MLE. The accuracy of various maximum likelihood estimations under the per-pixel regime across varying signal-to-background ratios. The performance gap increases as noise becomes more dominant.
• Figure 3: Information Sharing Pilot Study. The setup involves two autoencoders attempting to reconstruct the inputs while isolating the common features in the latent space. The reconstruction results and the corresponding feature distribution verify the claims about shared features. "PC $1$", "PC $2$" and "PC $3$" on the axes represent the $1^{\text{th}}$, $2^{\text{nd}}$ and $3^{\text{rd}}$ principal components.
• Figure 4: Overview of the SPLiDER Network. The proposed SPLiDER network consists of four main modules. Given several individual timestamp frames directly from a SPAD array, binary frames are generated via a thresholding process: a pixel is marked as 1 if a timestamp exists, and 0 otherwise. Next, depth and reflectivity features from several adjacent input frames are extracted using two feature extraction networks. These features are then warped together based on the optical flow between frames. Subsequently, a cross-model information fusion module is employed to enhance the feature sets of the respective modalities. This module identifies the uniquely observed features by the depth and reflectivity feature extraction mechanisms and fuses them cohesively. Warping and information sharing are performed at three resolution scales to capture both fine and coarse details, improving feature representation. Finally, the multi-scale reconstruction network simultaneously reconstructs depth and reflectivity.
• Figure 5: Functionality of the CCAM Module. The CCAM module identifies common features uniquely observed by each channel's feature extraction mechanism and shares them across channels. Due to noisy input, the feature set of the input images is incomplete compared to the feature set of the ground truths.

Theorems & Definitions (6)

• Theorem 1: Joint density of $M$ timestamps $\mathbf{t}_M$ Bar-David_1969vivek_2024_detection
• Corollary 1
• Corollary 2
• Corollary 3
• Theorem 2: CRLB comparison between the reflectivity estimators
• Lemma 1