Cross-View Referring Multi-Object Tracking

TL;DR

This work defines Cross-view Referring Multi-Object Tracking (CRMOT) to overcome single-view RMOT limitations by leveraging overlapping views for robust appearance capture and language-grounded identity tracking. It introduces the CRTrack benchmark (from DIVOTrack and CAMPUS) with 13 scenes, 82K frames, 344 objects, and 221 language descriptions, using invariant attribute-based annotations and GPT-4o-generated descriptions refined by humans, plus new CRMOT metrics CVRIDF1 and CVRMA. An end-to-end CRTracker baseline combines CrossMOT-inspired tracking with APTM-based language and attribute guidance, incorporating a prediction module that fuses trajectory-language signals to output CRMOT trajectories. Experiments show state-of-the-art performance on in-domain data and notable generalization to cross-domain settings, with detailed ablations confirming the efficacy of the prediction module. The work provides a practical, multi-view, language-guided tracking framework and a publicly available benchmark to advance CRMOT research.

Abstract

Referring Multi-Object Tracking (RMOT) is an important topic in the current tracking field. Its task form is to guide the tracker to track objects that match the language description. Current research mainly focuses on referring multi-object tracking under single-view, which refers to a view sequence or multiple unrelated view sequences. However, in the single-view, some appearances of objects are easily invisible, resulting in incorrect matching of objects with the language description. In this work, we propose a new task, called Cross-view Referring Multi-Object Tracking (CRMOT). It introduces the cross-view to obtain the appearances of objects from multiple views, avoiding the problem of the invisible appearances of objects in RMOT task. CRMOT is a more challenging task of accurately tracking the objects that match the language description and maintaining the identity consistency of objects in each cross-view. To advance CRMOT task, we construct a cross-view referring multi-object tracking benchmark based on CAMPUS and DIVOTrack datasets, named CRTrack. Specifically, it provides 13 different scenes and 221 language descriptions. Furthermore, we propose an end-to-end cross-view referring multi-object tracking method, named CRTracker. Extensive experiments on the CRTrack benchmark verify the effectiveness of our method. The dataset and code are available at https://github.com/chen-si-jia/CRMOT.

Figures (16)

• Figure 1: The difference between CRMOT and RMOT. The CRMOT task introduces the cross-view to obtain the appearances of objects from multiple views, avoiding the problem that the appearances of objects are easily invisible in the RMOT task.
• Figure 2: Language Description Annotation Pipeline.
• Figure 3: Word Cloud.
• Figure 4: Pipeline of CRTracker. It includes a detection head, a single-view Re-ID head, a cross-view Re-ID head, a full Re-ID head and APTM framework. The prediction module outputs the trajectories of objects that match the language description.
• Figure 5: Qualitative results of our proposed CRTracker method and other methods, including TransRMOT and TempRMOT, on the CRTrack benchmark's in-domain and cross-domain evaluations. The rows and columns represent the camera views and different methods, respectively. Red arrows indicate targets that are not correctly detected or matched. Other colored arrows represent correctly detected targets, with arrows of the same color indicating the same target.