VideoMultiAgents: A Multi-Agent Framework for Video Question Answering

TL;DR

VideoMultiAgents introduces a modular multi-agent framework for Video Question Answering that partitions reasoning into modality-specific Text, Video, and Graph agents, coordinated by an Organizer Agent. It adds Question-Guided Caption Generation to produce query-focused captions and uses scene graphs to encode temporal and causal relationships, improving reasoning over long-form videos. The approach achieves state-of-the-art zero-shot performance on NExT-QA, Intent-QA, and EgoSchema, demonstrating strong generalization without task-specific training. This architecture offers scalable, interpretable multimodal reasoning and can extend to additional modalities and tasks with minimal redesign.

Abstract

Video Question Answering (VQA) inherently relies on multimodal reasoning, integrating visual, temporal, and linguistic cues to achieve a deeper understanding of video content. However, many existing methods rely on feeding frame-level captions into a single model, making it difficult to adequately capture temporal and interactive contexts. To address this limitation, we introduce VideoMultiAgents, a framework that integrates specialized agents for vision, scene graph analysis, and text processing. It enhances video understanding leveraging complementary multimodal reasoning from independently operating agents. Our approach is also supplemented with a question-guided caption generation, which produces captions that highlight objects, actions, and temporal transitions directly relevant to a given query, thus improving the answer accuracy. Experimental results demonstrate that our method achieves state-of-the-art performance on Intent-QA (79.0%, +6.2% over previous SOTA), EgoSchema subset (75.4%, +3.4%), and NExT-QA (79.6%, +0.4%). The source code is available at https://github.com/PanasonicConnect/VideoMultiAgents.

Figures (3)

• Figure 1: Overview of VideoMultiAgents: Integrating Video, Captions, and Scene Graphs through Multi-Agent Collaboration for accurate Video Question Answering. Blue arrows indicate inputs and outputs to and from the VideoMultiAgents, while yellow arrows represent interactions and information exchanges between individual agents, coordinated by the Organizer Agent to determine the final answer.
• Figure 2: Detailed Architecture of a VideoMultiAgents for Video Question Answering: Each specialized agent independently analyzes its assigned modality—text, video, or scene graph—based on the input video and question, generating its own response. The Text Analysis Agent utilizes question-guided captions to extract key entities and actions. The Video Analysis Agent identifies objects, interactions, and temporal events. The Graph Analysis Agent constructs structured representations of object relationships and causal dynamics. The Organizer Agent integrates these independent responses to synthesize the final answer.
• Figure 3: Inference Process of VideoMultiAgents on Long Videos. This illustration demonstrates our system's analysis of an hour-long video, showcasing the collaborative yet independent reasoning of specialized agents. While the Video Analysis Agent identifies Option B as the answer, both Text and Graph Analysis Agents select Option D with supporting temporal evidence. The Organizer Agent synthesizes these perspectives, examining reasoning quality from each agent before concluding that Option D has stronger evidential support. This example highlights the framework's ability to resolve conflicting interpretations through structured multi-agent collaboration, resulting in accurate understanding even with extended video content.