Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach

Shiao Wang, Xiao Wang, Liye Jin, Bo Jiang, Lin Zhu, Lan Chen, Yonghong Tian, Bin Luo

TL;DR

The paper tackles low-latency visual object tracking in event camera data by introducing SFTrack, a Slow-Fast dual-head framework that jointly leverages high-temporal-resolution event streams and ViT backbones. It employs a two-stage training regime: stage-1 independently trains a slow, high-precision tracker and a lightweight fast tracker using graph-based event features and FlashAttention, and stage-2 uses supervised fine-tuning and knowledge distillation to fuse the two pathways while boosting the fast tracker. The slow tracker integrates multi-scale event graphs with a 12-layer FlashViT, while the fast tracker uses a pruned 6-layer ViT and a simple cross-view fusion to maintain millisecond-level latency, achieving up to 126 FPS on EventVOT and strong SR/PR/NPR gains across FE240hz, COESOT, and EventVOT. The approach demonstrates robust performance in resource-rich and resource-constrained environments, providing practical low-latency tracking for real-world applications such as robotics and autonomous systems, with insights into graph construction, fusion strategies, and distillation-based knowledge transfer.

Abstract

Existing tracking algorithms typically rely on low-frame-rate RGB cameras coupled with computationally intensive deep neural network architectures to achieve effective tracking. However, such frame-based methods inherently face challenges in achieving low-latency performance and often fail in resource-constrained environments. Visual object tracking using bio-inspired event cameras has emerged as a promising research direction in recent years, offering distinct advantages for low-latency applications. In this paper, we propose a novel Slow-Fast Tracking paradigm that flexibly adapts to different operational requirements, termed SFTrack. The proposed framework supports two complementary modes, i.e., a high-precision slow tracker for scenarios with sufficient computational resources, and an efficient fast tracker tailored for latency-aware, resource-constrained environments. Specifically, our framework first performs graph-based representation learning from high-temporal-resolution event streams, and then integrates the learned graph-structured information into two FlashAttention-based vision backbones, yielding the slow and fast trackers, respectively. The fast tracker achieves low latency through a lightweight network design and by producing multiple bounding box outputs in a single forward pass. Finally, we seamlessly combine both trackers via supervised fine-tuning and further enhance the fast tracker's performance through a knowledge distillation strategy. Extensive experiments on public benchmarks, including FE240, COESOT, and EventVOT, demonstrate the effectiveness and efficiency of our proposed method across different real-world scenarios. The source code has been released on https://github.com/Event-AHU/SlowFast_Event_Track.

Towards Low-Latency Event Stream-based Visual Object Tracking: A Slow-Fast Approach

TL;DR

The paper tackles low-latency visual object tracking in event camera data by introducing SFTrack, a Slow-Fast dual-head framework that jointly leverages high-temporal-resolution event streams and ViT backbones. It employs a two-stage training regime: stage-1 independently trains a slow, high-precision tracker and a lightweight fast tracker using graph-based event features and FlashAttention, and stage-2 uses supervised fine-tuning and knowledge distillation to fuse the two pathways while boosting the fast tracker. The slow tracker integrates multi-scale event graphs with a 12-layer FlashViT, while the fast tracker uses a pruned 6-layer ViT and a simple cross-view fusion to maintain millisecond-level latency, achieving up to 126 FPS on EventVOT and strong SR/PR/NPR gains across FE240hz, COESOT, and EventVOT. The approach demonstrates robust performance in resource-rich and resource-constrained environments, providing practical low-latency tracking for real-world applications such as robotics and autonomous systems, with insights into graph construction, fusion strategies, and distillation-based knowledge transfer.

Abstract

Existing tracking algorithms typically rely on low-frame-rate RGB cameras coupled with computationally intensive deep neural network architectures to achieve effective tracking. However, such frame-based methods inherently face challenges in achieving low-latency performance and often fail in resource-constrained environments. Visual object tracking using bio-inspired event cameras has emerged as a promising research direction in recent years, offering distinct advantages for low-latency applications. In this paper, we propose a novel Slow-Fast Tracking paradigm that flexibly adapts to different operational requirements, termed SFTrack. The proposed framework supports two complementary modes, i.e., a high-precision slow tracker for scenarios with sufficient computational resources, and an efficient fast tracker tailored for latency-aware, resource-constrained environments. Specifically, our framework first performs graph-based representation learning from high-temporal-resolution event streams, and then integrates the learned graph-structured information into two FlashAttention-based vision backbones, yielding the slow and fast trackers, respectively. The fast tracker achieves low latency through a lightweight network design and by producing multiple bounding box outputs in a single forward pass. Finally, we seamlessly combine both trackers via supervised fine-tuning and further enhance the fast tracker's performance through a knowledge distillation strategy. Extensive experiments on public benchmarks, including FE240, COESOT, and EventVOT, demonstrate the effectiveness and efficiency of our proposed method across different real-world scenarios. The source code has been released on https://github.com/Event-AHU/SlowFast_Event_Track.
Paper Structure (19 sections, 6 equations, 9 figures, 7 tables)

This paper contains 19 sections, 6 equations, 9 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Event Stream based Tracking
  4. Slow-Fast Learning
  5. Low-Latency Design
  6. Methodology
  7. Overview
  8. Input Representation
  9. Network Architecture
  10. Loss Function
  11. Experiments
  12. Datasets and Evaluation Metric
  13. Implementation Details
  14. Comparison on Public Benchmarks
  15. Ablation Study
  16. ...and 4 more sections

Figures (9)

  • Figure 1: Existing event-based tracking methods either adopt (a) a one-stream based tracking or (b) a Siamese network based tracking, both of which suffer from high latency. We propose (c) a dual-head tracking framework designed to achieve low-latency tracking in line with real-world requirements. (d) A comparison between the conventional frame-based high-latency tracker and the event stream-driven low-latency tracker.
  • Figure 2: Comparison of Accuracy and Tracking Speed with other Trackers.
  • Figure 3: An overview of our proposed Slow-Fast Framework for Event-based Tracking. It contains a slow tracker with a 12-layer flash-attention based ViT and a fast tracker with a 6-layer. In addition to the input event frames, both trackers are also fed with event points to optimize their tracking performance. Specifically, we first train the slow and fast tracker separately, enhancing their tracking performance through effective fusion with the event-based graph feature representation. Subsequently, by employing supervised fine-tuning and a knowledge distillation strategy, we not only achieve seamless integration between the two trackers but also significantly improve the performance of the fast tracker. Our Slow-Fast approach demonstrates flexible adaptability to real-world scenarios, whether under resource-rich or resource-constrained conditions.
  • Figure 4: Visualization of tracking results on the EventVOT dataset.
  • Figure 5: (a). Analysis of the number of event points. (b). Analysis of the number of GCN layers.
  • ...and 4 more figures