LaMOT: Language-Guided Multi-Object Tracking

TL;DR

This work defines Language-Guided MOT as a unified task that leverages natural language for multi-object tracking, combining open-vocabulary and referring-expression capabilities. It introduces LaMOT, the largest standardized benchmark for Vision-Language MOT (1,660 sequences, 1.67M frames, 18.9K trajectories across five scenarios) with high-quality textual annotations, and proposes LaMOTer, a simple yet effective baseline that fuses GroundingDINO-based detection with OC-SORT-based tracking. Comprehensive experiments and scenario analyses demonstrate competitive performance and provide insights into open-vocabulary language use and challenging video contexts. The dataset and baseline are intended to accelerate development of flexible, language-guided tracking systems with real-world applicability.

Abstract

Vision-Language MOT is a crucial tracking problem and has drawn increasing attention recently. It aims to track objects based on human language commands, replacing the traditional use of templates or pre-set information from training sets in conventional tracking tasks. Despite various efforts, a key challenge lies in the lack of a clear understanding of why language is used for tracking, which hinders further development in this field. In this paper, we address this challenge by introducing Language-Guided MOT, a unified task framework, along with a corresponding large-scale benchmark, termed LaMOT, which encompasses diverse scenarios and language descriptions. Specially, LaMOT comprises 1,660 sequences from 4 different datasets and aims to unify various Vision-Language MOT tasks while providing a standardized evaluation platform. To ensure high-quality annotations, we manually assign appropriate descriptive texts to each target in every video and conduct careful inspection and correction. To the best of our knowledge, LaMOT is the first benchmark dedicated to Language-Guided MOT. Additionally, we propose a simple yet effective tracker, termed LaMOTer. By establishing a unified task framework, providing challenging benchmarks, and offering insights for future algorithm design and evaluation, we expect to contribute to the advancement of research in Vision-Language MOT. We will release the data at https://github.com/Nathan-Li123/LaMOT.

Figures (5)

• Figure 1: The visualization of two main settings of Vision-Language MOT, i.e., open-vocabulary classname tracking and referring expression tracking. Moreover, Language-Guided MOT enables tracking with any form of language while possessing the ability to recognize open-vocabulary terms.
• Figure 2: (a) Representative examples from LaMOT, including sequences from five different scenarios. (b) Distribution of track counts from different scenarios, highlighting two scenarios from the same dataset dave2020tao. (c) Word cloud of the vocabulary used in LaMOT.
• Figure 3: Illustration of our proposed LaMOTer, which contains two components of vision-language detection and object tracking. "KF" stands for Kalman Filter kalman1960new.
• Figure 4: Qualitative results of LaMOTer on LaMOT. We observe that LaMOTer achieves satisfactory on all 5 scenarios. Each color represents a tracking trajectory.
• Figure 5: Difficulty comparison of different scenarios in LaMOT using different metric including HOTA (image(a)), AssA (image(b)), DetA (image(c)), and MOTA (image(d)).