Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LongVideoAgent: Multi-Agent Reasoning with Long Videos

Runtao Liu, Ziyi Liu, Jiaqi Tang, Yue Ma, Renjie Pi, Jipeng Zhang, Qifeng Chen

TL;DR

LongVideoAgent introduces a modular, agent-based approach to long-form video question answering, where a MasterAgent coordinates a GroundingAgent for temporal localization and a VisionAgent for targeted visual observations. The master is trained with reinforcement learning (GRPO) under a two-signal reward to encourage structured, concise reasoning and correct final answers, while grounding and vision modules remain frozen. The authors create episode-level benchmarks LongTVQA and LongTVQA+ and demonstrate state-of-the-art performance over non-agent baselines, with ablations validating the value of multi-agent coordination, visual grounding, and RL. This work advances scalable, interpretable long-form multimodal reasoning and highlights the potential of agentic planning with tool-based perception for complex video QA tasks.

Abstract

Recent advances in multimodal LLMs and systems that use tools for long-video QA point to the promise of reasoning over hour-long episodes. However, many methods still compress content into lossy summaries or rely on limited toolsets, weakening temporal grounding and missing fine-grained cues. We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations. The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation. This design helps the master agent focus on relevant clips via grounding, complements subtitles with visual detail, and yields interpretable trajectories. On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines. Experiments also show reinforcement learning further strengthens reasoning and planning for the trained agent. Code and data will be shared at https://longvideoagent.github.io/.

LongVideoAgent: Multi-Agent Reasoning with Long Videos

TL;DR

LongVideoAgent introduces a modular, agent-based approach to long-form video question answering, where a MasterAgent coordinates a GroundingAgent for temporal localization and a VisionAgent for targeted visual observations. The master is trained with reinforcement learning (GRPO) under a two-signal reward to encourage structured, concise reasoning and correct final answers, while grounding and vision modules remain frozen. The authors create episode-level benchmarks LongTVQA and LongTVQA+ and demonstrate state-of-the-art performance over non-agent baselines, with ablations validating the value of multi-agent coordination, visual grounding, and RL. This work advances scalable, interpretable long-form multimodal reasoning and highlights the potential of agentic planning with tool-based perception for complex video QA tasks.

Abstract

Recent advances in multimodal LLMs and systems that use tools for long-video QA point to the promise of reasoning over hour-long episodes. However, many methods still compress content into lossy summaries or rely on limited toolsets, weakening temporal grounding and missing fine-grained cues. We propose a multi-agent framework in which a master LLM coordinates a grounding agent to localize question-relevant segments and a vision agent to extract targeted textual observations. The master agent plans with a step limit, and is trained with reinforcement learning to encourage concise, correct, and efficient multi-agent cooperation. This design helps the master agent focus on relevant clips via grounding, complements subtitles with visual detail, and yields interpretable trajectories. On our proposed LongTVQA and LongTVQA+ which are episode-level datasets aggregated from TVQA/TVQA+, our multi-agent system significantly outperforms strong non-agent baselines. Experiments also show reinforcement learning further strengthens reasoning and planning for the trained agent. Code and data will be shared at https://longvideoagent.github.io/.
Paper Structure (25 sections, 2 figures, 5 tables)

This paper contains 25 sections, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Video Question Answering
  4. LLM Agents
  5. Multi-Modal LLMs
  6. Method
  7. Multi-agent System Framework
  8. Master agent behavior and training.
  9. Grounding agent.
  10. Vision agent.
  11. Reinforcement Learning for LongVideoAgent
  12. Trajectory.
  13. Rewards.
  14. Objective and optimization.
  15. Experiments
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Traditional single-pass MLLMs that ingest entire long videos in one context—typically (may through heavy downsampling and compression) often miss crucial evidence and produce wrong answers, whereas LongVideoAgent conducts multi-agent, multi-round, and multimodal reasoning to extract sparse, task-relevant cues and answer correctly.
  • Figure 2: Architecture of LongVideoAgent. A MasterAgent runs for up to $K$ rounds, collaborating with a GroundingAgent to localize relevant clips from videos and a VisionAgent to read fine-grained cues from the localized frames. Evidence accumulates until the MasterAgent feels confident to answer the user.