Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning

Yichang Xu, Gaowen Liu, Ramana Rao Kompella, Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Ling Liu

Abstract

This paper presents a multi-agent perception-action exploration alliance, dubbed A4VL, for efficient long-video reasoning. A4VL operates in a multi-round perception-action exploration loop with a selection of VLM agents. In each round, the team of agents performs video question-answer (VideoQA) via perception exploration followed by action exploration. During perception exploration, each agent learns to extract query-specific perception clue(s) from a few sampled frames and performs clue-based alignment to find the video block(s) that are most relevant to the query-specific event. During action exploration, A4VL performs video reasoning in three steps: (1) each agent produces its initial answer with rational, (2) all agents collaboratively scores one another through cross-reviews and relevance ranking, and (3) based on whether a satisfactory consensus is reached, the decision is made either to start a new round of perception-action deliberation by pruning (e.g., filtering out the lowest performing agent) and re-staging (e.g., new-clue and matching block based perception-action exploration), or to conclude by producing its final answer. The integration of the multi-agent alliance through multi-round perception-action exploration, coupled with event-driven partitioning and cue-guided block alignment, enables A4VL to effectively scale to real world long videos while preserving high quality video reasoning. Evaluation Results on five popular VideoQA benchmarks show that A4VL outperforms 18 existing representative VLMs and 10 recent methods optimized for long-video reasoning, while achieving significantly lower inference latency. Our code is released at https://github.com/git-disl/A4VL.

A Multi-Agent Perception-Action Alliance for Efficient Long Video Reasoning

Abstract

This paper presents a multi-agent perception-action exploration alliance, dubbed A4VL, for efficient long-video reasoning. A4VL operates in a multi-round perception-action exploration loop with a selection of VLM agents. In each round, the team of agents performs video question-answer (VideoQA) via perception exploration followed by action exploration. During perception exploration, each agent learns to extract query-specific perception clue(s) from a few sampled frames and performs clue-based alignment to find the video block(s) that are most relevant to the query-specific event. During action exploration, A4VL performs video reasoning in three steps: (1) each agent produces its initial answer with rational, (2) all agents collaboratively scores one another through cross-reviews and relevance ranking, and (3) based on whether a satisfactory consensus is reached, the decision is made either to start a new round of perception-action deliberation by pruning (e.g., filtering out the lowest performing agent) and re-staging (e.g., new-clue and matching block based perception-action exploration), or to conclude by producing its final answer. The integration of the multi-agent alliance through multi-round perception-action exploration, coupled with event-driven partitioning and cue-guided block alignment, enables A4VL to effectively scale to real world long videos while preserving high quality video reasoning. Evaluation Results on five popular VideoQA benchmarks show that A4VL outperforms 18 existing representative VLMs and 10 recent methods optimized for long-video reasoning, while achieving significantly lower inference latency. Our code is released at https://github.com/git-disl/A4VL.
Paper Structure (25 sections, 45 equations, 9 figures, 7 tables, 2 algorithms)

This paper contains 25 sections, 45 equations, 9 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Design Overview
  4. Perception Exploration
  5. Action Exploration
  6. Experiment
  7. Experimental Settings
  8. Results
  9. Ablation Study
  10. Conclusion
  11. A4VL Algorithm
  12. Additional Visualizations
  13. Failure Cases
  14. Event-Based Partitioning
  15. Sampled frames and notation.
  16. ...and 10 more sections

Figures (9)

  • Figure 1: Responses from A4VL and three of the strongest closed-sourced models.
  • Figure 2: Overview of A4VL architecture.
  • Figure 3: A4VL agent teaming workflow.
  • Figure 4: Example of A4VL on EgoSchema. Here all agents have wrong answers at the beginning, but later one of them turns to the correct answer and other agents are pruned.
  • Figure 5: Relationship between accuracy and maximum number of rounds.
  • ...and 4 more figures