ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model

TL;DR

Vision-Language tracking lags behind SoTA visual trackers due to reliance on manual language and semantic gaps. The authors propose ChatTracker, integrating Multimodal Large Language Models via a Reflection-based Prompt Optimization (RPO) and a semantic grounding framework to produce accurate foreground/background descriptions and proposals. RPO refines language outputs through iterative grounding and prompt updates, while a GVLM-based Semantic Tracking module grounds descriptions to generate foreground and background proposals; a Foreground Verification module computes a final score $s^{i}=s^{i}_{fore}\cdot(1-s^{i}_{back})$ to select the tracking result. Experiments on LaSOT, TrackingNet, and TNL2K show competitive or state-of-the-art performance and demonstrate cross-model generalization, underscoring the practical potential of knowledge-rich tracking.

Abstract

Visual object tracking aims to locate a targeted object in a video sequence based on an initial bounding box. Recently, Vision-Language~(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this paper, we propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions and enhance tracking performance. To this end, we propose a novel reflection-based prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, a simple yet effective VL tracking framework is proposed and can be easily integrated as a plug-and-play module to boost the performance of both VL and visual trackers. Experimental results show that our proposed ChatTracker achieves a performance comparable to existing methods.

Figures (3)

• Figure 1: Comparison of different text generation methods. (a) shows manual descriptions and GPT-4V generated descriptions of the tracking target, which are both sub-optimal for tracking. (b) illustrates the generation method used in ChatTracker.
• Figure 2: Overall framework of the proposed algorithm. It primarily consists of three parts: A Reflection-based Prompt Optimization Module designed to generate descriptions of both the foreground and background elements to track accurately, a Semantic Tracking Module tasked with creating region proposals for these areas based on the generated descriptions, and a Foreground Verification Module that utilizes these region proposals to select the most precise tracking results. Note that the values in the figure are for visualization and may not match the actual implementation exactly.
• Figure 3: Illustrations of prompt optimization in a dialogue scenario. Each set shows the initial manual annotation, the subsequent prompts generated by the LLM, and the final optimized prompt that successfully guided the Vision-Language tracker to the target.