Motion-Aware Transformer for Multi-Object Tracking

TL;DR

DETR-based MOT often suffers from query collisions between track and detection queries, hindering stable association. The authors propose Motion-Aware Transformer (MATR) with a Motion Attention Transformer (MAT) module that explicitly predicts motion to update track queries before decoding, supervised by a dense trajectory loss on full sequences. Baseline enhancements plus MAT yield substantial gains, achieving state-of-the-art results on DanceTrack, SportsMOT, and BDD100k with modest computational overhead and strong ablations clarifying the benefit of motion modeling. The work demonstrates that incorporating motion dynamics into end-to-end Transformer-based MOT provides a simple, effective path to robust, scalable tracking across diverse scenes.

Abstract

Multi-object tracking (MOT) in videos remains challenging due to complex object motions and crowded scenes. Recent DETR-based frameworks offer end-to-end solutions but typically process detection and tracking queries jointly within a single Transformer Decoder layer, leading to conflicts and degraded association accuracy. We introduce the Motion-Aware Transformer (MATR), which explicitly predicts object movements across frames to update track queries in advance. By reducing query collisions, MATR enables more consistent training and improves both detection and association. Extensive experiments on DanceTrack, SportsMOT, and BDD100k show that MATR delivers significant gains across standard metrics. On DanceTrack, MATR improves HOTA by more than 9 points over MOTR without additional data and reaches a new state-of-the-art score of 71.3 with supplementary data. MATR also achieves state-of-the-art results on SportsMOT (72.2 HOTA) and BDD100k (54.7 mTETA, 41.6 mHOTA) without relying on external datasets. These results demonstrate that explicitly modeling motion within end-to-end Transformers offers a simple yet highly effective approach to advancing multi-object tracking.

Figures (6)

• Figure 1: Illustration of query collisions in MOTR.
• Figure 2: Distribution of track query distances ($1 - \text{IoU}$) for different methods. We collected these distances from collision points during loss computation. Each point represents the distance between a track query’s current position and its corresponding ground truth. The dashed lines indicate the average distances. For reference, the gray dashed line shows the average distance of detect queries assigned by Hungarian matching.
• Figure 3: Overview of the MATR architecture. Unlike MOTR, MATR leverages features from the Transformer Encoder ("Memory") to update track queries $U_{\text{trk}}^{i}$ at time $i$. The input frame is denoted by $I^{i}$, and the sequence of frames by $S$. $Q_{\det}$ represents learnable detection queries, and $Q_{\text{trk}}^{i}$ are the tracking results at time $i$. The Motion-Aware Transformer (MAT) module is supervised by a motion loss and trained jointly with the Decoder. By predicting future positions, MAT updates track query features and positional embeddings in advance.
• Figure 4: Design of the proposed Motion-Aware Transformer (MAT) module. An additional Transformer Decoder predicts and updates the track queries from the previous frame $Q_{\text{trk}}^{t-1}$, generating refined queries $U_{\text{trk}}^{t-1}$. The MAT module is supervised with a trajectory loss and trained jointly with the main Decoder outputs.
• Figure 5: Illustration of the MATR loss design. Left: MAT loss, where dashed lines denote intermediate modules. Trajectory loss is computed across an entire sequence of length $S$, using L1 distance between predicted and ground-truth trajectories. Right: overview of MATR outputs and losses during training.