Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning

Chenglin Li, Qianglong Chen, Feng Han, Yikun Wang, Xingxi Yin, Yan Gong, Ruilin Li, Yin Zhang, Jiaqi Wang

TL;DR

VideoThinker tackles long-form video understanding by enabling adaptive tool-guided reasoning over video content. It trains an end-to-end VideoLLM entirely on synthetic tool-interaction trajectories generated in caption space, grounding reasoning back to frames through frame-level perception tools. The approach introduces two agentic tools—Temporal Retrieval and Temporal Zoom—and a confidence-gated controller to trigger deeper reasoning when needed. Across four long-form benchmarks, VideoThinker outperforms caption-only baselines and matches or approaches state-of-the-art LLM-based agents while remaining a lightweight 7B model. This demonstrates that tool-augmented synthetic data and adaptive retrieval-zoom strategies can effectively scale long-form video understanding.

Abstract

Long-form video understanding remains a fundamental challenge for current Video Large Language Models. Most existing models rely on static reasoning over uniformly sampled frames, which weakens temporal localization and leads to substantial information loss in long videos. Agentic tools such as temporal retrieval, spatial zoom, and temporal zoom offer a natural way to overcome these limitations by enabling adaptive exploration of key moments. However, constructing agentic video understanding data requires models that already possess strong long-form video comprehension, creating a circular dependency. We address this challenge with VideoThinker, an agentic Video Large Language Model trained entirely on synthetic tool interaction trajectories. Our key idea is to convert videos into rich captions and employ a powerful agentic language model to generate multi-step tool use sequences in caption space. These trajectories are subsequently grounded back to video by replacing captions with the corresponding frames, yielding a large-scale interleaved video and tool reasoning dataset without requiring any long-form understanding from the underlying model. Training on this synthetic agentic dataset equips VideoThinker with dynamic reasoning capabilities, adaptive temporal exploration, and multi-step tool use. Remarkably, VideoThinker significantly outperforms both caption-only language model agents and strong video model baselines across long-video benchmarks, demonstrating the effectiveness of tool augmented synthetic data and adaptive retrieval and zoom reasoning for long-form video understanding.

VideoThinker: Building Agentic VideoLLMs with LLM-Guided Tool Reasoning

TL;DR

VideoThinker tackles long-form video understanding by enabling adaptive tool-guided reasoning over video content. It trains an end-to-end VideoLLM entirely on synthetic tool-interaction trajectories generated in caption space, grounding reasoning back to frames through frame-level perception tools. The approach introduces two agentic tools—Temporal Retrieval and Temporal Zoom—and a confidence-gated controller to trigger deeper reasoning when needed. Across four long-form benchmarks, VideoThinker outperforms caption-only baselines and matches or approaches state-of-the-art LLM-based agents while remaining a lightweight 7B model. This demonstrates that tool-augmented synthetic data and adaptive retrieval-zoom strategies can effectively scale long-form video understanding.

Abstract

Long-form video understanding remains a fundamental challenge for current Video Large Language Models. Most existing models rely on static reasoning over uniformly sampled frames, which weakens temporal localization and leads to substantial information loss in long videos. Agentic tools such as temporal retrieval, spatial zoom, and temporal zoom offer a natural way to overcome these limitations by enabling adaptive exploration of key moments. However, constructing agentic video understanding data requires models that already possess strong long-form video comprehension, creating a circular dependency. We address this challenge with VideoThinker, an agentic Video Large Language Model trained entirely on synthetic tool interaction trajectories. Our key idea is to convert videos into rich captions and employ a powerful agentic language model to generate multi-step tool use sequences in caption space. These trajectories are subsequently grounded back to video by replacing captions with the corresponding frames, yielding a large-scale interleaved video and tool reasoning dataset without requiring any long-form understanding from the underlying model. Training on this synthetic agentic dataset equips VideoThinker with dynamic reasoning capabilities, adaptive temporal exploration, and multi-step tool use. Remarkably, VideoThinker significantly outperforms both caption-only language model agents and strong video model baselines across long-video benchmarks, demonstrating the effectiveness of tool augmented synthetic data and adaptive retrieval and zoom reasoning for long-form video understanding.
Paper Structure (29 sections, 2 equations, 16 figures, 4 tables, 2 algorithms)

This paper contains 29 sections, 2 equations, 16 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Agentic LLMs for Video Understanding
  4. Multimodal reasoning with MLLMs
  5. Long-form VideoQA and Video-LLMs
  6. Method
  7. Tools Design
  8. Temporal Retrieval
  9. Temporal Zoom
  10. Data Synthesis with Agentic LLMs
  11. Multimodal Tool-Reasoning Training
  12. Adaptive Video Reasoning
  13. Experiments
  14. Experimental Setup
  15. Benchmarks
  16. ...and 14 more sections

Figures (16)

  • Figure 1: Comparison of VideoThinker with VideoLLMs and LLM agents. VideoThinker excels at interleaved video reasoning on long videos, using agentic tools to iteratively perceive and reason over video frames step by step.
  • Figure 2: VideoThinker integrates retrieval and zoom tools for multi-turn reasoning. LLMs use caption_zoom to generate reasoning data from videos, which is later replaced by frame_zoom and video frames to build agentic video understanding CoTs.
  • Figure 3: The prompt designed to enable VideoLLM to serve as a caption generation tool.
  • Figure 4: Confidence–accuracy relationship. The analysis is conducted on samples from VideoMME (2.7k), LongVideoBench (1.3k), and LVBench (1.5k).
  • Figure 5: Accuracy on LongVideoBench and VideoMME across different video durations. Short: $<2$ min; Medium: $2$--$15$ min; Long: $>15$ min.
  • ...and 11 more figures