Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Search

TL;DR

The paper tackles long-form video understanding under limited context by proposing HAVEN, which grounds semantics through audiovisual entity cohesion and a four-level hierarchical index spanning global, scene, segment, and entity. It employs offline construction of this hierarchy and an agentic Think–Act–Observe loop with multi-granularity tools to enable coherent, multi-turn reasoning over long videos. Empirical results on LVBench and other benchmarks show state-of-the-art performance (e.g., overall LVBench accuracy of 84.1% and strong reasoning gains), with ablations highlighting the critical role of hierarchical indexing and multimodal entity integration. The work demonstrates that structured, multimodal reasoning with an offline hierarchy can deliver scalable, contextually coherent understanding of long-form videos, with practical impact for AI assistants, search, and analytics on extended video content.

Abstract

Long video understanding presents significant challenges for vision-language models due to extremely long context windows. Existing solutions relying on naive chunking strategies with retrieval-augmented generation, typically suffer from information fragmentation and a loss of global coherence. We present HAVEN, a unified framework for long-video understanding that enables coherent and comprehensive reasoning by integrating audiovisual entity cohesion and hierarchical video indexing with agentic search. First, we preserve semantic consistency by integrating entity-level representations across visual and auditory streams, while organizing content into a structured hierarchy spanning global summary, scene, segment, and entity levels. Then we employ an agentic search mechanism to enable dynamic retrieval and reasoning across these layers, facilitating coherent narrative reconstruction and fine-grained entity tracking. Extensive experiments demonstrate that our method achieves good temporal coherence, entity consistency, and retrieval efficiency, establishing a new state-of-the-art with an overall accuracy of 84.1% on LVBench. Notably, it achieves outstanding performance in the challenging reasoning category, reaching 80.1%. These results highlight the effectiveness of structured, multimodal reasoning for comprehensive and context-consistent understanding of long-form videos.

Figures (6)

• Figure 1: The proposed hierarchical video indexing with audiovisual entity cohesion and agentic search.
• Figure 2: Overview of our framework. Left: hierarchical database construction. Right: agentic reasoning. The reasoning LLM calls tools iteratively to collect information and answer the question.
• Figure 3: Comparison of accuracy and efficiency on six categories. "Acc" and "iter" denote accuracy and the average number of reasoning iterations, respectively.
• Figure 4: Case study on different reasoning chains with tool calls.
• Figure 5: Cases: speaker identity for entity tracking.