Unsupervised Spike Depth Estimation via Cross-modality Cross-domain Knowledge Transfer

TL;DR

This work tackles unsupervised spike depth estimation by bridging RGB and neuromorphic spike modalities through BiCross, a cross-modality cross-domain framework. It introduces Coarse-to-Fine Knowledge Distillation ($L_{CFKD}$) to transfer RGB semantics to sparse spike data and a Self-Correcting Teacher-Student scheme ($L_{SCP}$,$L_{SCA}$) to address domain shift without accumulating errors. Across four scenarios (Synthetic to Real, Extreme Weather, Scene Changing, Real Spike), BiCross achieves state-of-the-art performance among RGB-based unsupervised methods and demonstrates robust depth estimation with limited labeled spike data. The approach enables leveraging abundant RGB annotations to enhance spike-depth tasks and provides new spike-depth datasets for ongoing research, with practical implications for high-speed autonomous systems.

Abstract

Neuromorphic spike data, an upcoming modality with high temporal resolution, has shown promising potential in autonomous driving by mitigating the challenges posed by high-velocity motion blur. However, training the spike depth estimation network holds significant challenges in two aspects: sparse spatial information for pixel-wise tasks and difficulties in achieving paired depth labels for temporally intensive spike streams. Therefore, we introduce open-source RGB data to support spike depth estimation, leveraging its annotations and spatial information. The inherent differences in modalities and data distribution make it challenging to directly apply transfer learning from open-source RGB to target spike data. To this end, we propose a cross-modality cross-domain (BiCross) framework to realize unsupervised spike depth estimation by introducing simulated mediate source spike data. Specifically, we design a Coarse-to-Fine Knowledge Distillation (CFKD) approach to facilitate comprehensive cross-modality knowledge transfer while preserving the unique strengths of both modalities, utilizing a spike-oriented uncertainty scheme. Then, we propose a Self-Correcting Teacher-Student (SCTS) mechanism to screen out reliable pixel-wise pseudo labels and ease the domain shift of the student model, which avoids error accumulation in target spike data. To verify the effectiveness of BiCross, we conduct extensive experiments on four scenarios, including Synthetic to Real, Extreme Weather, Scene Changing, and Real Spike. Our method achieves state-of-the-art (SOTA) performances, compared with RGB-oriented unsupervised depth estimation methods. Code and dataset: https://github.com/Theia-4869/BiCross

Figures (4)

• Figure 1: demonstrates the process of BiCross and the distinct properties of RGB and spike modality. While RGB data contains sufficient semantic information, spike data is limited in its feature space due to sparse spatial information. However, spike data shows strength in depth estimation on dynamic objects, owning to its dense temporal resolution.
• Figure 2: The BiCross framework composes the cross-modality and cross-domain phases. The first two rows demonstrate cross-modality learning, we propose CFKD with the spike-oriented uncertainty filter (part a) to transfer sufficient knowledge from RGB teacher model to spike student model under the source domain. The last two rows show the cross-domain learning, we introduce a Self-Correcting Teacher-Student (SCTS) scheme in which the teacher model utilizes the pixel-wise uncertainty method to screen out reliable depth estimation (part b1) and the student model adopts global-level alignment (part b2) to correct domain shift. The uncertainty estimation approach is conducted at the model output layer, following the two phases. CFKD and SCTS jointly contribute to achieving unsupervised spike depth estimation.
• Figure 3: The spike data is sent into the spike encoder as shown in part (a). Part (b) shows the decoder and prediction heads of the network. In part (c), we show the mechanism of spike-oriented uncertainty filter in CFKD, it selects the distilled knowledge which should be reliable in the RGB feature and demanded in the sparse spike feature. Red circle.1 represents the objectives of cross-modality learning, including coarse (CKD) and fine-level (FKD) knowledge distillation. Red circle.2 contains self-correcting pseudo-label (SCP) and self-correcting alignment (SCA) in the cross domain phase.
• Figure 4: Baseline represents the output of $Pre_{src}$. Uncertainty Map is predicted by our method in unsupervised cross-domain phase.