SOTA: Spike-Navigated Optimal TrAnsport Saliency Region Detection in Composite-bias Videos

TL;DR

Spike-navigated Optimal Transport Saliency Region Detection (SOTA) addresses real-world saliency detection in high-temporal-resolution spike camera videos by mitigating composite noise and time-varying lighting domain gaps. The method introduces Spike-based Micro-Debias (SM) for micro-temporal refinement via multi-scale spiking features and Spike-based Global Debias (SG) via Kantorovich-based OT to align spike saliency distributions with real-image distributions, trained in an adversarial framework. The two components are jointly optimized with a max–min objective and an EM-distance penalty, achieving state-of-the-art results on Spike-DAVIS and SVS across single-step and multi-step settings while maintaining energy efficiency, as evidenced by quantitative gains in MAE and F-measures and favorable visualizations. This work demonstrates that integrating neuromorphic temporal modeling with OT-based global distribution alignment can robustly detect salient regions under challenging motion and lighting conditions, offering practical impact for surveillance and real-time video analysis using spike cameras.

Abstract

Existing saliency detection methods struggle in real-world scenarios due to motion blur and occlusions. In contrast, spike cameras, with their high temporal resolution, significantly enhance visual saliency maps. However, the composite noise inherent to spike camera imaging introduces discontinuities in saliency detection. Low-quality samples further distort model predictions, leading to saliency bias. To address these challenges, we propose Spike-navigated Optimal TrAnsport Saliency Region Detection (SOTA), a framework that leverages the strengths of spike cameras while mitigating biases in both spatial and temporal dimensions. Our method introduces Spike-based Micro-debias (SM) to capture subtle frame-to-frame variations and preserve critical details, even under minimal scene or lighting changes. Additionally, Spike-based Global-debias (SG) refines predictions by reducing inconsistencies across diverse conditions. Extensive experiments on real and synthetic datasets demonstrate that SOTA outperforms existing methods by eliminating composite noise bias. Our code and dataset will be released at https://github.com/lwxfight/sota.

Figures (9)

• Figure 1: Imaging Comparison. (a) RGB vision cameras, which are affected by time gaps. (b) Spike camera imaging, where (b1) shows the traditional reconstruction method, influenced by environmental noise, leading to inaccurate saliency maps. (b2) presents the optimized reconstruction from a spatiotemporal perspective, resulting in a more accurate saliency map.
• Figure 2: Temporal and Spatial Debias. Temporal debias captures subtle changes by exploring deep feature connections, while spatial debias constructs an OT map to minimize the distance between the spike saliency distribution and the real image distribution.
• Figure 3: Composite Noise Challenges in Spike Cameras. (a) Traditional imaging forms the foundation of spike cameras. (b) Ideally, continuous spike streams and photons ensure smooth object motion. However, real-world light variations and background interference often result in missing photons, causing biased pixel representations.
• Figure 4: Motivation of SOTA. The left side shows variations in light conditions across samples, with different colored lines representing individual samples. Green boxes indicate correct predictions, while red boxes highlight errors.
• Figure 5: Overview of the Proposed SOTA. The two networks are optimized iteratively: $T$-Net generates temporal saliency maps with micro-detail debias, while $F$-Net refines global spatial debias. Together, they enhance motion associations in the spatiotemporal saliency map and model long-term dependencies.