Z-GMOT: Zero-shot Generic Multiple Object Tracking

TL;DR

The paper addresses the limitations of traditional MOT and one-shot GMOT by introducing a zero-shot Generic Multiple Object Tracking framework, Z-GMOT, that operates without training data or predefined categories. It leverages a new Vision-Language-based detector, iGLIP, and a novel association strategy, MA-SORT, to track unseen generic objects using natural language prompts and appearance-motion fusion. A new Referring GMOT dataset, with Refer-GMOT40 and Refer-Animal, provides textual attribute descriptions to support zero-shot tracking, and extensive experiments demonstrate that Z-GMOT achieves competitive or superior performance to fully supervised MOT methods on challenging benchmarks while maintaining open-set capabilities. The work also showcases generalizability to MOT tasks, offering a practical, scalable approach for diverse tracking scenarios in surveillance, robotics, and wildlife monitoring, with released code and models to facilitate adoption and further research.

Abstract

Despite recent significant progress, Multi-Object Tracking (MOT) faces limitations such as reliance on prior knowledge and predefined categories and struggles with unseen objects. To address these issues, Generic Multiple Object Tracking (GMOT) has emerged as an alternative approach, requiring less prior information. However, current GMOT methods often rely on initial bounding boxes and struggle to handle variations in factors such as viewpoint, lighting, occlusion, and scale, among others. Our contributions commence with the introduction of the \textit{Referring GMOT dataset} a collection of videos, each accompanied by detailed textual descriptions of their attributes. Subsequently, we propose $\mathtt{Z-GMOT}$, a cutting-edge tracking solution capable of tracking objects from \textit{never-seen categories} without the need of initial bounding boxes or predefined categories. Within our $\mathtt{Z-GMOT}$ framework, we introduce two novel components: (i) $\mathtt{iGLIP}$, an improved Grounded language-image pretraining, for accurately detecting unseen objects with specific characteristics. (ii) $\mathtt{MA-SORT}$, a novel object association approach that adeptly integrates motion and appearance-based matching strategies to tackle the complex task of tracking objects with high similarity. Our contributions are benchmarked through extensive experiments conducted on the Referring GMOT dataset for GMOT task. Additionally, to assess the generalizability of the proposed $\mathtt{Z-GMOT}$, we conduct ablation studies on the DanceTrack and MOT20 datasets for the MOT task. Our dataset, code, and models are released at: https://fsoft-aic.github.io/Z-GMOT.

Figures (7)

• Figure 1: High-level comparison between our $\mathtt{Z-GMOT}$ with conventional MOT and one-shot Generic MOT (OS-GMOT) for the task of tracking athletes in red uniforms on a running track. $1^{st}$ row: MOT, being a fully-supervised method, using YOLOX (trained on COCO) and OC-SORT (trained on DanceTrack) attempts to detect and track all people in the scene with high False Positive (FPs). $2^{nd}$ row: OS-GMOT is based on an initial bounding box and utilizes an MOT tracker (e.g. OS-SORT in this case). While reducing the number of FPs, OS-GMOT heavily relies on the initial bounding box, leading to variations in results with different bounding boxes and a high number of False Negatives (FNs). $3^{rd}$ row: our $\mathtt{Z-GMOT}$ including: (i) $\mathtt{iGLIP}$ effectively detects objects without the need for prior training or initial bounding boxes, and (ii) $\mathtt{MA-SORT}$ efficiently associates objects with high visual similarity.
• Figure 2: Examples of data annotation structure.
• Figure 3: Limitation 1 of GLIP: Sensitive to threshold selection. With slightly different thresholds $t=0.6$ v.s. $t=0.66$, GLIP produces different results with high FPs (left) and high FNs (right). Note that GLIP uses prompt "red car") in both results.
• Figure 4: Limitation 2 of GLIP: With the same $t$. ($t = 0.64$) and the same prompt "red car", the results vary when applied to two similar input images.
• Figure 5: Network architecture of $\mathtt{iGLIP}$, which inputs an image $I$, a general prompt $T_g$ (e.g. "ball"), and a specific prompt $T_s$ (e.g. "red ball"). $\mathtt{iGLIP}$ includes a QGM module to eliminate FPs generated from the general prompt.