HAD: Hierarchical Asymmetric Distillation to Bridge Spatio-Temporal Gaps in Event-Based Object Tracking

TL;DR

HAD addresses the spatio-temporal asymmetry between RGB frames and event streams in single-object tracking by introducing a hierarchical distillation framework. It explicitly aligns temporal dynamics through a GRU-based Temporal Alignment module and enforces geometry-aware spatial alignment via Spatial-Aligned Optimal Transport, enabling an event-only student to inherit bimodal teacher knowledge. Across EventVOT, COESOT, and VisEvent, HAD achieves state-of-the-art or competitive performance for unimodal event tracking while preserving efficiency, and ablations validate the necessity of each alignment component. The approach offers a practical path to robust cross-modal perception by leveraging rich RGB information during training while maintaining modality independence at inference, with potential extensions to additional modalities in future work.

Abstract

RGB cameras excel at capturing rich texture details with high spatial resolution, whereas event cameras offer exceptional temporal resolution and a high dynamic range (HDR). Leveraging their complementary strengths can substantially enhance object tracking under challenging conditions, such as high-speed motion, HDR environments, and dynamic background interference. However, a significant spatio-temporal asymmetry exists between these two modalities due to their fundamentally different imaging mechanisms, hindering effective multi-modal integration. To address this issue, we propose {Hierarchical Asymmetric Distillation} (HAD), a multi-modal knowledge distillation framework that explicitly models and mitigates spatio-temporal asymmetries. Specifically, HAD proposes a hierarchical alignment strategy that minimizes information loss while maintaining the student network's computational efficiency and parameter compactness. Extensive experiments demonstrate that HAD consistently outperforms state-of-the-art methods, and comprehensive ablation studies further validate the effectiveness and necessity of each designed component. The code will be released soon.

Figures (10)

• Figure 1: Motivation of HAD. (a) RGB cameras capture rich texture details under standard conditions, whereas event cameras encode rapid motion information under extreme environments. Multi-modal fusion leverages these complementary advantages. (b) Effective fusion requires explicitly addressing the inherent temporal and spatial asymmetry between RGB frames and event streams.
• Figure 2: Sampling mechanisms of RGB and event cameras. (a) Event cameras operate based on a differential circuit principle, asynchronously triggering ON/OFF events at each pixel in response to local light-intensity changes. (b) RGB cameras employ a Bayer filter arrangement and perform synchronous sampling at a fixed frame rate, as used in traditional image sensors.
• Figure 3: Analysis of representative sequences on COESOT. Two representative sequences are separated by a dashed line. Each analysis panel (arranged from left to right) contains four complementary subfigures: (1) Comparative visualization of RGB ground-truth keyframes and corresponding event frames; (2) Dynamic-edge intersection-over-union (IoU) between RGB frames and event streams (blue); (3) Optical-flow alignment errors for both modalities (yellow); and (4) Texture-contrast comparison between event streams (green) and RGB frames (red).
• Figure 4: Overview of HAD. A bimodal teacher network (RGB + event) guides an event-only student through Hierarchical Asymmetric Distillation. The framework integrates a GRU-based temporal alignment module to synchronize asynchronous feature sequences and an entropic optimal transport-based spatial alignment module to align response distributions across modalities.
• Figure 5: Radar charts of PR metrics on EventVOT. Each axis corresponds to a specific tracking challenge, illustrating performance across different attributes. Zooming in reveals finer performance differences among competing methods.