Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking

TL;DR

The paper presents SeqTrack, a sequence-to-sequence visual tracking framework that casts bounding-box prediction as autoregressive token generation using a compact encoder–decoder transformer, eliminating complex heads and bespoke losses. Building on this, SeqTrackv2 introduces a unified interface and task-prompt tokens to support multi-modal tracking (RGB+Depth, RGB+Thermal, RGB+Event, RGB+Language) with a shared model. Empirical results across 14 benchmarks show SeqTrack and SeqTrackv2 achieve competitive to state-of-the-art performance, with SeqTrackv2 delivering strong multi-modal improvements while maintaining efficiency. The approach simplifies cross-modal tracking and offers a scalable path toward unified multimodal perception in tracking systems.

Abstract

In this paper, we introduce a new sequence-to-sequence learning framework for RGB-based and multi-modal object tracking. First, we present SeqTrack for RGB-based tracking. It casts visual tracking as a sequence generation task, forecasting object bounding boxes in an autoregressive manner. This differs from previous trackers, which depend on the design of intricate head networks, such as classification and regression heads. SeqTrack employs a basic encoder-decoder transformer architecture. The encoder utilizes a bidirectional transformer for feature extraction, while the decoder generates bounding box sequences autoregressively using a causal transformer. The loss function is a plain cross-entropy. Second, we introduce SeqTrackv2, a unified sequence-to-sequence framework for multi-modal tracking tasks. Expanding upon SeqTrack, SeqTrackv2 integrates a unified interface for auxiliary modalities and a set of task-prompt tokens to specify the task. This enables it to manage multi-modal tracking tasks using a unified model and parameter set. This sequence learning paradigm not only simplifies the tracking framework, but also showcases superior performance across 14 challenging benchmarks spanning five single- and multi-modal tracking tasks. The code and models are available at https://github.com/chenxin-dlut/SeqTrackv2.

Figures (9)

• Figure 1: Comparison of tracking frameworks. (a) The framework with object classification head and bounding box regression head. (b) The framework with corner prediction heads. (c) Sequence-to-sequence tracking framework without complicated head networks. (d) Sequence-to-sequence framework for multi-modal tracking.
• Figure 2: (a) Architecture of the proposed SeqTrack. The key component is an encoder-decoder transformer. The encoder extracts visual features from input video frames. The causal decoder autoregressively generates the sequence of the bounding box tokens using the extracted features. (b) Detailed transformer block in the causal decoder. The input embeddings interact in a casual manner through a masked multi-head attention mechanism. The visual feature is incorporated into the decoder via a multi-head attention layer.
• Figure 3: (a) Illustration of the causal attention mask in the decoder using a maximum sequence length of 5 tokens. An orange cell at row $i$ and column $j$ indicates that the attention mechanism is allowed to attend to the $j$th input token, when producing the $i$th output token. (b) Illustration of the input and target sequences. Similar to autoregressive language modeling 2017Attention, the input sequence is the target sequence with one position offset.
• Figure 4: Architecture of the proposed SeqTrackv2. Expanding on SeqTrack, SeqTrackv2 integrates a unified interface and task-prompt tokens. This integration consolidates diverse multi-modal tracking tasks into a unified model and parameter set.
• Figure 5: Details of the proposed unified interface. (a) Various auxiliary modalities are convert into a sequence format. (b) The auxiliary-modal sequence is integrated into the visual features of the encoder.