Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMTrack: Semantic Multi-Object Tracking with Multi-modal Large Language Models

Pan Liao, Feng Yang, Di Wu, Jinwen Yu, Yuhua Zhu, Wenhui Zhao

TL;DR

The paper addresses the gap in multi-object tracking where traditional systems excel at localization but lack semantic reasoning about object behavior. It introduces LLMTrack, an end-to-end framework that fuses perception (Grounding DINO) with reasoning (LLaVA-OneVision) through a Spatio-Temporal Fusion Module, enabling semantic outputs such as instance descriptions and scene interactions. A progressive three-stage training pipeline (Visual Alignment, Temporal Fine-tuning, Semantic Injection via LoRA) ensures efficient adaptation of a large language model to tracking tasks. Evaluated on BenSMOT, LLMTrack achieves state-of-the-art results in both tracking and semantic understanding, including improved instance descriptions, interaction recognition, and video summarization, while maintaining tracking stability and robustness for downstream cognitive perception applications.

Abstract

Traditional Multi-Object Tracking (MOT) systems have achieved remarkable precision in localization and association, effectively answering \textit{where} and \textit{who}. However, they often function as autistic observers, capable of tracing geometric paths but blind to the semantic \textit{what} and \textit{why} behind object behaviors. To bridge the gap between geometric perception and cognitive reasoning, we propose \textbf{LLMTrack}, a novel end-to-end framework for Semantic Multi-Object Tracking (SMOT). We adopt a bionic design philosophy that decouples strong localization from deep understanding, utilizing Grounding DINO as the eyes and the LLaVA-OneVision multimodal large model as the brain. We introduce a Spatio-Temporal Fusion Module that aggregates instance-level interaction features and video-level contexts, enabling the Large Language Model (LLM) to comprehend complex trajectories. Furthermore, we design a progressive three-stage training strategy, Visual Alignment, Temporal Fine-tuning, and Semantic Injection via LoRA to efficiently adapt the massive model to the tracking domain. Extensive experiments on the BenSMOT benchmark demonstrate that LLMTrack achieves state-of-the-art performance, significantly outperforming existing methods in instance description, interaction recognition, and video summarization while maintaining robust tracking stability.

LLMTrack: Semantic Multi-Object Tracking with Multi-modal Large Language Models

TL;DR

The paper addresses the gap in multi-object tracking where traditional systems excel at localization but lack semantic reasoning about object behavior. It introduces LLMTrack, an end-to-end framework that fuses perception (Grounding DINO) with reasoning (LLaVA-OneVision) through a Spatio-Temporal Fusion Module, enabling semantic outputs such as instance descriptions and scene interactions. A progressive three-stage training pipeline (Visual Alignment, Temporal Fine-tuning, Semantic Injection via LoRA) ensures efficient adaptation of a large language model to tracking tasks. Evaluated on BenSMOT, LLMTrack achieves state-of-the-art results in both tracking and semantic understanding, including improved instance descriptions, interaction recognition, and video summarization, while maintaining tracking stability and robustness for downstream cognitive perception applications.

Abstract

Traditional Multi-Object Tracking (MOT) systems have achieved remarkable precision in localization and association, effectively answering \textit{where} and \textit{who}. However, they often function as autistic observers, capable of tracing geometric paths but blind to the semantic \textit{what} and \textit{why} behind object behaviors. To bridge the gap between geometric perception and cognitive reasoning, we propose \textbf{LLMTrack}, a novel end-to-end framework for Semantic Multi-Object Tracking (SMOT). We adopt a bionic design philosophy that decouples strong localization from deep understanding, utilizing Grounding DINO as the eyes and the LLaVA-OneVision multimodal large model as the brain. We introduce a Spatio-Temporal Fusion Module that aggregates instance-level interaction features and video-level contexts, enabling the Large Language Model (LLM) to comprehend complex trajectories. Furthermore, we design a progressive three-stage training strategy, Visual Alignment, Temporal Fine-tuning, and Semantic Injection via LoRA to efficiently adapt the massive model to the tracking domain. Extensive experiments on the BenSMOT benchmark demonstrate that LLMTrack achieves state-of-the-art performance, significantly outperforming existing methods in instance description, interaction recognition, and video summarization while maintaining robust tracking stability.
Paper Structure (15 sections, 3 equations, 3 figures, 3 tables)

This paper contains 15 sections, 3 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From Geometric Perception to Semantic Cognition
  4. Vision-Language Models and LLaVA-OneVision
  5. Method
  6. Architecture Overview
  7. Spatio-Temporal Fusion Module
  8. Instance Fusion: Adaptive Temporal Attention.
  9. Video Fusion: Recursive Context Aggregation.
  10. Progressive Three-Stage Training
  11. Experiments
  12. Setup and Metrics
  13. Quantitative Results
  14. Ablation Studies
  15. Conclusion

Figures (3)

  • Figure 1: Overall architecture of LLMTrack. It consists of a visual frontend (Grounding DINO), a Spatio-Temporal Fusion Module, and a reasoning backend (LLaVA-OneVision).
  • Figure 2: Detailed structure of the Spatio-Temporal Fusion Module, featuring instance-level interaction modeling and video-level context recursion.
  • Figure 3: The progressive three-stage training pipeline: (1) Visual Alignment, (2) Temporal Fine-tuning, and (3) Semantic Injection.