AR-MOT: Autoregressive Multi-object Tracking

TL;DR

AR-MOT introduces an autoregressive MOT paradigm by formulating multi-object tracking as next-token prediction within a multimodal LLM. The framework combines an Image Tokenizer for global context, an Object Tokenizer for region-level perception, and Region-Aware Alignment plus Temporal Memory Fusion to maintain identity over time without task-specific heads. Through training on MOT17 and DanceTrack, AR-MOT achieves competitive performance and demonstrates strong extensibility to integrate new modalities via simple changes to the output sequence. This work lays groundwork for general, instruction-driven MOT systems that can leverage broad LLM capabilities for richer, flexible tracking in diverse scenarios.

Abstract

As multi-object tracking (MOT) tasks continue to evolve toward more general and multi-modal scenarios, the rigid and task-specific architectures of existing MOT methods increasingly hinder their applicability across diverse tasks and limit flexibility in adapting to new tracking formulations. Most approaches rely on fixed output heads and bespoke tracking pipelines, making them difficult to extend to more complex or instruction-driven tasks. To address these limitations, we propose AR-MOT, a novel autoregressive paradigm that formulates MOT as a sequence generation task within a large language model (LLM) framework. This design enables the model to output structured results through flexible sequence construction, without requiring any task-specific heads. To enhance region-level visual perception, we introduce an Object Tokenizer based on a pretrained detector. To mitigate the misalignment between global and regional features, we propose a Region-Aware Alignment (RAA) module, and to support long-term tracking, we design a Temporal Memory Fusion (TMF) module that caches historical object tokens. AR-MOT offers strong potential for extensibility, as new modalities or instructions can be integrated by simply modifying the output sequence format without altering the model architecture. Extensive experiments on MOT17 and DanceTrack validate the feasibility of our approach, achieving performance comparable to state-of-the-art methods while laying the foundation for more general and flexible MOT systems.

Figures (5)

• Figure 1: Comparison of different MOT paradigms. (a) Tracking by Detection paradiam. (b) Tracking by Query paradiam. (c) Tracking by autoregressively generation.
• Figure 2: Framework of AR-MOT. The proposed model consists of a Large Language Model along with two main modules (a)(b), and two additional modules (c)(d). (a) Image Tokenizer. Image Tokenizer is responsible for capturing global visual semantics from low-resolution images. It uses a SigLIP-ViT backbone to extract feature sequences, which are then projected into the LLM space through a vision adapter to produce image tokens. (b) Object Tokenizer. Object Tokenizer focuses on region-level perception by applying a DETR-based detector to high-resolution images. It generates object-specific features and projects them into the LLM space using an object adapter to obtain object tokens. (c) Region-Aware Alignment Module. RAA performs semantic alignment by projecting the bounding boxes from the Object Tokenizer onto the image tokenizer space to retrieve the corresponding image tokens, which are fused with object tokens to produce aligned object representations. (d) Temporal Memory Fusion Module. TMF maintains a memory of informative object tokens from earlier frames and dynamically integrates them into the current tokens. This design extends the temporal context while maintaining computational efficiency.
• Figure 3: Temporal Memory Fusion module. Upon completing tracking for the current frame, the corresponding object tokens are retained as reference for the next frame. The TMF module incorporates historical object tokens into the current ones, providing long-term temporal context for subsequent tracking.
• Figure 4: Impact of the $\tau_{loss}$ Parameter on DetA and AssA Metrics. Preserving more historical frames with larger $\tau_{loss}$ has limited impact on DetA, while AssA changes more significantly, highlighting the importance of temporal history for association modeling.
• Figure 5: Sequence-wise Visualization Results at $\tau_{loss}=10$. TCM boosts performance across all sequences, with notable enhancements on specific ones(e.g., dancetrack0097, dancetrack0079)