Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding

Teng Fu, Mengyang Zhao, Ke Niu, Kaixin Peng, Bin Li

TL;DR

OmniPT introduces a unified LVLM-based pedestrian tracker that performs MOT, referring tracking, and semantic understanding. It advances a four-stage training pipeline—GRPO-based output standardization, Mid Training, supervised fine-tuning, and final RL refinement—to map tracking tasks to LVLM capabilities and produce fixed-format outputs. Empirical results across DanceTrack, BenSMOT, Refer-KITTI-V2, and CRTrack demonstrate state-of-the-art performance across MOT, RMOT, CRMOT, and SMOT, with substantial gains in HOTA and captioning metrics. The work showcases LVLMs as a versatile, interactive engine for joint tracking and semantic reasoning, with implications for open-vocabulary and reference-guided tracking, while noting limitations in crowded scenes and generalization that invite future LVLM improvements.

Abstract

LVLMs have been shown to perform excellently in image-level tasks such as VQA and caption. However, in many instance-level tasks, such as visual grounding and object detection, LVLMs still show performance gaps compared to previous expert models. Meanwhile, although pedestrian tracking is a classical task, there have been a number of new topics in combining object tracking and natural language, such as Referring MOT, Cross-view Referring MOT, and Semantic MOT. These tasks emphasize that models should understand the tracked object at an advanced semantic level, which is exactly where LVLMs excel. In this paper, we propose a new unified Pedestrian Tracking framework, namely OmniPT, which can track, track based on reference and generate semantic understanding of tracked objects interactively. We address two issues: how to model the tracking task into a task that foundation models can perform, and how to make the model output formatted answers. To this end, we implement a training phase consisting of RL-Mid Training-SFT-RL. Based on the pre-trained weights of the LVLM, we first perform a simple RL phase to enable the model to output fixed and supervisable bounding box format. Subsequently, we conduct a mid-training phase using a large number of pedestrian-related datasets. Finally, we perform supervised fine-tuning on several pedestrian tracking datasets, and then carry out another RL phase to improve the model's tracking performance and enhance its ability to follow instructions. We conduct experiments on tracking benchmarks and the experimental results demonstrate that the proposed method can perform better than the previous methods.

OmniPT: Unleashing the Potential of Large Vision Language Models for Pedestrian Tracking and Understanding

TL;DR

OmniPT introduces a unified LVLM-based pedestrian tracker that performs MOT, referring tracking, and semantic understanding. It advances a four-stage training pipeline—GRPO-based output standardization, Mid Training, supervised fine-tuning, and final RL refinement—to map tracking tasks to LVLM capabilities and produce fixed-format outputs. Empirical results across DanceTrack, BenSMOT, Refer-KITTI-V2, and CRTrack demonstrate state-of-the-art performance across MOT, RMOT, CRMOT, and SMOT, with substantial gains in HOTA and captioning metrics. The work showcases LVLMs as a versatile, interactive engine for joint tracking and semantic reasoning, with implications for open-vocabulary and reference-guided tracking, while noting limitations in crowded scenes and generalization that invite future LVLM improvements.

Abstract

LVLMs have been shown to perform excellently in image-level tasks such as VQA and caption. However, in many instance-level tasks, such as visual grounding and object detection, LVLMs still show performance gaps compared to previous expert models. Meanwhile, although pedestrian tracking is a classical task, there have been a number of new topics in combining object tracking and natural language, such as Referring MOT, Cross-view Referring MOT, and Semantic MOT. These tasks emphasize that models should understand the tracked object at an advanced semantic level, which is exactly where LVLMs excel. In this paper, we propose a new unified Pedestrian Tracking framework, namely OmniPT, which can track, track based on reference and generate semantic understanding of tracked objects interactively. We address two issues: how to model the tracking task into a task that foundation models can perform, and how to make the model output formatted answers. To this end, we implement a training phase consisting of RL-Mid Training-SFT-RL. Based on the pre-trained weights of the LVLM, we first perform a simple RL phase to enable the model to output fixed and supervisable bounding box format. Subsequently, we conduct a mid-training phase using a large number of pedestrian-related datasets. Finally, we perform supervised fine-tuning on several pedestrian tracking datasets, and then carry out another RL phase to improve the model's tracking performance and enhance its ability to follow instructions. We conduct experiments on tracking benchmarks and the experimental results demonstrate that the proposed method can perform better than the previous methods.

Paper Structure

This paper contains 17 sections, 2 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Language-based Tracking Tasks
  4. Foundation model in MOT
  5. Methodology
  6. Group Relative Policy Optimization
  7. Mid Training
  8. Supervised Fine-tuning
  9. Inference
  10. Experiments
  11. Datasets and Metrics
  12. Implementation Details
  13. Main Results
  14. Ablation Studies
  15. Limitation
  16. ...and 2 more sections

Figures (4)

  • Figure 1: OmniPT can perform traditional object tracking; it can also perform object-specific tracking based on given references and can perform semantic understanding of the tracked objects as well as the whole video.
  • Figure 2: The format of the inputs and outputs for the different tasks our method can process.
  • Figure 3: The model architecture of our baseline LVLM and our two-stage fine-tuning strategy for the OmniPT.
  • Figure 4: The format of the queries and responses for the different tasks during supervised fine-tuning. RMOT and CRMOT are based on a design for single object tracking.