RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph

TL;DR

RAVU tackles the challenge of long-form video understanding by constructing a spatio-temporal graph as a persistent memory and performing compositional reasoning over this graph to retrieve query-relevant segments. The method decouples memory generation from retrieval, using LMMs to build expressive per-frame graphs and an LLM to fuse them into a coherent spatio-temporal memory, while a set of predefined reasoning functions execute stepwise queries on the graph. This approach enables effective multi-hop reasoning and object tracking with a small retrieved frame budget (5–10), achieving competitive or superior results on NExT-QA and EgoSchema compared to state-of-the-art baselines. The work demonstrates the practical impact of graph-based long-term memory for scalable and accurate video understanding, and it provides a framework that can be applied to existing LMMs with minimal fine-tuning.

Abstract

Comprehending long videos remains a significant challenge for Large Multi-modal Models (LMMs). Current LMMs struggle to process even minutes to hours videos due to their lack of explicit memory and retrieval mechanisms. To address this limitation, we propose RAVU (Retrieval Augmented Video Understanding), a novel framework for video understanding enhanced by retrieval with compositional reasoning over a spatio-temporal graph. We construct a graph representation of the video, capturing both spatial and temporal relationships between entities. This graph serves as a long-term memory, allowing us to track objects and their actions across time. To answer complex queries, we decompose the queries into a sequence of reasoning steps and execute these steps on the graph, retrieving relevant key information. Our approach enables more accurate understanding of long videos, particularly for queries that require multi-hop reasoning and tracking objects across frames. Our approach demonstrate superior performances with limited retrieved frames (5-10) compared with other SOTA methods and baselines on two major video QA datasets, NExT-QA and EgoSchema.

Figures (4)

• Figure 1: Overview of RAVU, memorizing a video as a spatio-temporal graph and retrieving key relevant parts by reasoning over the graph with a given query.
• Figure 2: An entire pipeline of our RAVU consisting two core components: spatio-temporal graph generation (memory) and compositional reasoning (retrieval). These memory and retrieval mechanisms form the core of our proposed framework. Through the reasoning over the graph, the relevant frames can be identified and they are fed into the LMM to get a final answer for the given question.
• Figure 3: Spatio-temporal graph generation pipeline. Firstly, we detect entities in each frame and connect the detected entities across the frames, followed by LMM generates expressive descriptions for the frame and the entities with bounding box annotation. Then, we use LLM to construct the graph from them. The entity-wise events in the video are also generated to capture the segments of the video.
• Figure 4: An example of our frame retrieval process. We execute reasoning steps sequentially and concatenate frames from each step.