Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VPTracker: Global Vision-Language Tracking via Visual Prompt and MLLM

Jingchao Wang, Kaiwen Zhou, Zhijian Wu, Kunhua Ji, Dingjiang Huang, Yefeng Zheng

TL;DR

VPTracker introduces the first global vision–language tracking framework built on Multimodal Large Language Models (MLLMs). It integrates a location-aware visual prompting mechanism that injects spatial priors derived from the target's previous state, enabling robust global search while suppressing distractors. Evaluations on TNL2K and TNLLT show improved robustness, target re-identification, and long-term stability, suggesting that MLLMs can serve as strong backbones for vision–language tracking when guided by spatial priors. This work opens avenues for deeper integration of cross-modal reasoning and temporal localization in tracking systems.

Abstract

Vision-Language Tracking aims to continuously localize objects described by a visual template and a language description. Existing methods, however, are typically limited to local search, making them prone to failures under viewpoint changes, occlusions, and rapid target movements. In this work, we introduce the first global tracking framework based on Multimodal Large Language Models (VPTracker), exploiting their powerful semantic reasoning to locate targets across the entire image space. While global search improves robustness and reduces drift, it also introduces distractions from visually or semantically similar objects. To address this, we propose a location-aware visual prompting mechanism that incorporates spatial priors into the MLLM. Specifically, we construct a region-level prompt based on the target's previous location, enabling the model to prioritize region-level recognition and resort to global inference only when necessary. This design retains the advantages of global tracking while effectively suppressing interference from distracting visual content. Extensive experiments show that our approach significantly enhances tracking stability and target disambiguation under challenging scenarios, opening a new avenue for integrating MLLMs into visual tracking. Code is available at https://github.com/jcwang0602/VPTracker.

VPTracker: Global Vision-Language Tracking via Visual Prompt and MLLM

TL;DR

VPTracker introduces the first global vision–language tracking framework built on Multimodal Large Language Models (MLLMs). It integrates a location-aware visual prompting mechanism that injects spatial priors derived from the target's previous state, enabling robust global search while suppressing distractors. Evaluations on TNL2K and TNLLT show improved robustness, target re-identification, and long-term stability, suggesting that MLLMs can serve as strong backbones for vision–language tracking when guided by spatial priors. This work opens avenues for deeper integration of cross-modal reasoning and temporal localization in tracking systems.

Abstract

Vision-Language Tracking aims to continuously localize objects described by a visual template and a language description. Existing methods, however, are typically limited to local search, making them prone to failures under viewpoint changes, occlusions, and rapid target movements. In this work, we introduce the first global tracking framework based on Multimodal Large Language Models (VPTracker), exploiting their powerful semantic reasoning to locate targets across the entire image space. While global search improves robustness and reduces drift, it also introduces distractions from visually or semantically similar objects. To address this, we propose a location-aware visual prompting mechanism that incorporates spatial priors into the MLLM. Specifically, we construct a region-level prompt based on the target's previous location, enabling the model to prioritize region-level recognition and resort to global inference only when necessary. This design retains the advantages of global tracking while effectively suppressing interference from distracting visual content. Extensive experiments show that our approach significantly enhances tracking stability and target disambiguation under challenging scenarios, opening a new avenue for integrating MLLMs into visual tracking. Code is available at https://github.com/jcwang0602/VPTracker.
Paper Structure (18 sections, 2 equations, 4 figures, 3 tables)

This paper contains 18 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Object Tracking
  4. Multimodal Large Language Models
  5. Method
  6. Problem Formulation
  7. Global Tracking via MLLMs
  8. Visual Prompt based on Location-aware
  9. Experiment
  10. Implementation Details
  11. Dataset
  12. Training and Inference
  13. Comparison with State-of-the-art Methods
  14. TNLLT
  15. TNL2K
  16. ...and 3 more sections

Figures (4)

  • Figure 1: An overview of our proposed VPTracker.
  • Figure 2: Prompt used for Vision-Language Tracking.
  • Figure 3: Visual comparison of tracking results with and without Visual Prompt
  • Figure 4: Visual comparison of tracking results with and without Visual Prompt