FARTrack: Fast Autoregressive Visual Tracking with High Performance

TL;DR

FARTrack addresses the speed-performance bottleneck in visual tracking on edge devices by introducing a fast autoregressive, multi-template framework. It couples Task-Specific Self-Distillation, which compresses depth while preserving trajectory information through layer-wise distillation of trajectory tokens, with Inter-frame Autoregressive Sparsification, a sequence-level template pruning strategy that operates across frames without extra runtime overhead. The model uses a Transformer encoder to integrate visual, trajectory, and coordinate tokens in a unified vocabulary, enabling temporally coherent tracking with a real-time inference footprint. On GOT-10k and other benchmarks, FARTrack achieves strong speed while maintaining competitive AO, with fastest variants reaching up to 343 FPS on GPU and 121 FPS on CPU, demonstrating practical applicability for edge deployment.

Abstract

Inference speed and tracking performance are two critical evaluation metrics in the field of visual tracking. However, high-performance trackers often suffer from slow processing speeds, making them impractical for deployment on resource-constrained devices. To alleviate this issue, we propose FARTrack, a Fast Auto-Regressive Tracking framework. Since autoregression emphasizes the temporal nature of the trajectory sequence, it can maintain high performance while achieving efficient execution across various devices. FARTrack introduces Task-Specific Self-Distillation and Inter-frame Autoregressive Sparsification, designed from the perspectives of shallow-yet-accurate distillation and redundant-to-essential token optimization, respectively. Task-Specific Self-Distillation achieves model compression by distilling task-specific tokens layer by layer, enhancing the model's inference speed while avoiding suboptimal manual teacher-student layer pairs assignments. Meanwhile, Inter-frame Autoregressive Sparsification sequentially condenses multiple templates, avoiding additional runtime overhead while learning a temporally-global optimal sparsification strategy. FARTrack demonstrates outstanding speed and competitive performance. It delivers an AO of 70.6% on GOT-10k in real-time. Beyond, our fastest model achieves a speed of 343 FPS on the GPU and 121 FPS on the CPU.

Figures (8)

• Figure 1: Overview. (a) Comparison of our Task-Specific Self-Distillation and Classical Cross-Layer Distillation. (b) Inter-frame Autoregressive Sparsification for Multi-templates.
• Figure 2: FARTrack vs. Other Trackers: Performance-Speed Trade-off. Comparison of our FARTrack with the state-of-the-art trackers on GOT-10k in terms of tracking speed (horizontal axis) on GPU and AO performence (vertical axis). The diameter of the circle is proportional to the ratio of the model's speed to its performance. FARTracknaco significantly surpasses existing trackers in both tracking performance and inference speed.
• Figure 3: FARTrack Framework. FARTrack is a fast, high-performance multi-template autoregressive framework, comprising two key components: Task-Specific Self-Distillation for model compression and Inter-frame Autoregressive Sparsification for template sequences.
• Figure 4: Layer-by-layer distillation accuracy curve.
• Figure 5: Layer-wise cross-attention visualization. search: Search region and template. layer 0-14: Trajectory sequences to search cross-attention maps at each layer.