TP-GMOT: Tracking Generic Multiple Object by Textual Prompt with Motion-Appearance Cost (MAC) SORT

TL;DR

The paper tackles open-world GMOT by removing predefined-category priors and enabling tracking of generic objects via natural language prompts. It introduces Refer-GMOT, a dataset with fine-grained textual annotations, and the TP-GMOT framework consisting of TP-OD (iGDINO) and MAC-SORT for robust detection and data association. iGDINO enhances open-vocabulary detection with an Include-Exclude strategy and long-short memory, while MAC-SORT fuses motion and adaptive appearance cues with a novel cost formulation to handle highly similar objects. Experiments on Refer-GMOT and MOT benchmarks show strong performance gains and good generalization, with discussions on limitations and future directions, including prompt engineering and exploring other VLMs.

Abstract

While Multi-Object Tracking (MOT) has made substantial advancements, it is limited by heavy reliance on prior knowledge and limited to predefined categories. In contrast, Generic Multiple Object Tracking (GMOT), tracking multiple objects with similar appearance, requires less prior information about the targets but faces challenges with variants like viewpoint, lighting, occlusion, and resolution. Our contributions commence with the introduction of the \textbf{\text{Refer-GMOT dataset}} a collection of videos, each accompanied by fine-grained textual descriptions of their attributes. Subsequently, we introduce a novel text prompt-based open-vocabulary GMOT framework, called \textbf{\text{TP-GMOT}}, which can track never-seen object categories with zero training examples. Within \text{TP-GMOT} framework, we introduce two novel components: (i) {\textbf{\text{TP-OD}}, an object detection by a textual prompt}, for accurately detecting unseen objects with specific characteristics. (ii) Motion-Appearance Cost SORT \textbf{\text{MAC-SORT}}, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking multiple generic objects with high similarity. Our contributions are benchmarked on the \text{Refer-GMOT} dataset for GMOT task. Additionally, to assess the generalizability of the proposed \text{TP-GMOT} framework and the effectiveness of \text{MAC-SORT} tracker, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models will be publicly available at: https://fsoft-aic.github.io/TP-GMOT

Figures (4)

• Figure 1: Comparison between our TP-GMOT and the state-of-the-art GMOT approach, Visual Prompt-GMOT (VP-GMOT). While VP-GMOT requires an initial bounding box as input, our TP-GMOT accepts a natural language description as its input. Left: (a) VP-GMOT with two different initial bounding boxes and a MOT tracker (e.g., OC-SORT). Right: Our TP-GMOT introduces TP-OD for object detection with a natural language query and MAC-SORT for object association. The $1^{st}$ row shows the first video frame with initialization. The $2^{nd}$ and $3^{rd}$ rows display object detection at frames $\mathtt{t_1}$ and $\mathtt{t_2}$. The $4^{th}$ and $5^{th}$ rows show object association at frames $\mathtt{t_1}$ and $\mathtt{t_2}$, respectively.
• Figure 2: Examples of data annotation structure in our Refer-GMOT.
• Figure 3: An illustration of our proposed TP-OD for detecting objects using input prompt. It comprises two modules of IE strategy and LSM mechanism to eliminate FPs from pre-trained VLM.
• Figure 4: Comparison between apply LSM mechanism and not to eliminate FPs from pre-trained VLM.