Learning Association via Track-Detection Matching for Multi-Object Tracking

TL;DR

TDLP reframes multi-object tracking as per-frame track–detection link prediction, enabling a learning-based association that preserves the efficiency of tracking-by-detection while approaching end-to-end performance. It introduces a transformer-based, multi-modal architecture that fuses geometric, appearance, and pose cues, and uses a bipartite link predictor to score track–detection pairs for online Hungarian assignment. Across DanceTrack, SportsMOT, SoccerNet, MOT17, and BEE24, TDLP achieves state-of-the-art or near-state-of-the-art performance, with particularly strong gains in non-linear motion scenarios; even with only bounding-box features, TDLP can outperform many heuristic trackers. A detailed ablation shows link-prediction superiority over metric learning for heterogeneous features and analyzes robustness to detector failures. The work highlights a practical, modular approach that bridges efficient tracking-by-detection and expensive end-to-end methods, with future directions toward reducing quadratic costs and exploring longer temporal context.

Abstract

Multi-object tracking aims to maintain object identities over time by associating detections across video frames. Two dominant paradigms exist in literature: tracking-by-detection methods, which are computationally efficient but rely on handcrafted association heuristics, and end-to-end approaches, which learn association from data at the cost of higher computational complexity. We propose Track-Detection Link Prediction (TDLP), a tracking-by-detection method that performs per-frame association via link prediction between tracks and detections, i.e., by predicting the correct continuation of each track at every frame. TDLP is architecturally designed primarily for geometric features such as bounding boxes, while optionally incorporating additional cues, including pose and appearance. Unlike heuristic-based methods, TDLP learns association directly from data without handcrafted rules, while remaining modular and computationally efficient compared to end-to-end trackers. Extensive experiments on multiple benchmarks demonstrate that TDLP consistently surpasses state-of-the-art performance across both tracking-by-detection and end-to-end methods. Finally, we provide a detailed analysis comparing link prediction with metric learning-based association and show that link prediction is more effective, particularly when handling heterogeneous features such as detection bounding boxes. Our code is available at \href{https://github.com/Robotmurlock/TDLP}{https://github.com/Robotmurlock/TDLP}.

Figures (4)

• Figure 1: High-level overview of the proposed association pipeline. Encoded track histories and current detections are linked via link prediction and associated using bipartite matching.
• Figure 2: Architecture overview of the proposed TDLP method. Tracks and detections first undergo a geometric feature transformation, while optional additional cues (e.g., pose keypoints and appearance) are extracted using dedicated models. The static encoder processes per-frame detections from both current detections and track histories, while a motion encoder extracts track-specific temporal features (e.g., velocity). Track features are aggregated over time using a temporal encoder transformer. The resulting track and detection embeddings are fused across modalities and refined by an object-interaction encoder transformer. Finally, a link prediction head outputs association probabilities for each track–detection pair.
• Figure 3: Comparison of CTDP and TDLP across different bounding-box features (X, Y, W, H, C) measured by HOTA on SportsMOT validation set.
• Figure 4: Pass/fail matrices for CTDP and TDLP. Dots indicate correct outcomes; shaded cells (X) mark failures leading to ID switches. See Appendix \ref{['appendix:additional_results']}.