Vgent: Graph-based Retrieval-Reasoning-Augmented Generation For Long Video Understanding

TL;DR

This work tackles long-video understanding with LVLMs by addressing context-window limitations. It introduces Vgent, a graph-based retrieval-reasoning-augmented generation framework that offline constructs a video knowledge graph and preserves temporal dependencies via entity links. It adds a structured reasoning step to verify retrieved clips and aggregate information across clips before generation. On three long-video benchmarks across seven open LVLMs, Vgent yields improvements of $3.0\%\sim 5.4\%$ over base models and up to $8.6\%$ over state-of-the-art video RAG, underscoring its effectiveness and practicality.

Abstract

Understanding and reasoning over long videos pose significant challenges for large video language models (LVLMs) due to the difficulty in processing intensive video tokens beyond context window and retaining long-term sequential information. Retrieval-Augmented Generation (RAG) has demonstrated effectiveness in processing long context for Large Language Models (LLMs); however, applying RAG to long video faces challenges such as disrupted temporal dependencies and inclusion of irrelevant information that can hinder accurate reasoning. To address these limitations, we propose Vgent, a novel graph-based retrieval-reasoning-augmented generation framework to enhance LVLMs for long video understanding. Our approach introduces two key innovations: (i) It represents videos by structured graphs with semantic relationships across video clips preserved to improve retrieval effectiveness. (ii) It introduces an intermediate reasoning step to mitigate the reasoning limitation of LVLMs, which leverages structured verification to reduce retrieval noise and facilitate the explicit aggregation of relevant information across clips, resulting in more accurate and context-aware responses. We comprehensively evaluate our framework with various open-source LVLMs on three long-video understanding benchmarks. Our approach yielded an overall performance improvement of $3.0\%\sim 5.4\%$ over base models on MLVU, and outperformed state-of-the-art video RAG methods by $8.6\%$. Our code is publicly available at https://xiaoqian-shen.github.io/Vgent.

Figures (9)

• Figure 1: Overview of our graph-based retrieval-reasoning-augmented generation framework. Each video clip is represented as a node within a graph, interconnected through shared entities. This graph representation enables effective retrieval of relevant clips based on node connections, followed by an intermediate reasoning step to refine retrievals and aggregate over multimodal context for accurate generation.
• Figure 2: Pipeline of Vgent, a novel framework for long-context video understanding in the proposed graph-based retrieval-reasoning-augmented generation paradigm. It consists of four key stages: (1) Offline video graph construction (Section \ref{['sec:graph']}): Builds a video graph by extracting knowledge from long videos. (2) Graph-based retrieval (Section \ref{['sec:retrieve']}): Retrieves relevant clips based on keywords extracted from the user query. (3) Structured reasoning (Section \ref{['sec:reason']}): Refines clips using structured queries and aggregates information. (4) Multimodal augmented generation (Section \ref{['sec:generation']}): Combines refined clips and reasoning results to generate the final response.
• Figure 3: A qualitative example illustrates our graph-based retrieval-reasoning approach, which connects relevant video clips via shared entities. While the model initially fails to correctly identify the action of opening the laptop, misled by hallucinations from hard negatives. However, adding an intermediate reasoning step that validates each retrieved node through structured subqueries enables the model to generate the accurate response.
• Figure 4: Ablation studies. Left: retrieval embedding. Middle: number of retrieval $N$ before SR. Right: ablation on retrieval threshold $\tau$.
• Figure 5: A qualitative example illustrates our graph-based retrieval-reasoning approach.