Agentic Very Long Video Understanding

TL;DR

This work tackles the challenge of very long video understanding for always-on personal AI by introducing EGAgent, an agentic framework built around temporally annotated entity scene graphs. EGAgent combines a planning-based decision maker with specialized retrievers for visual frames, audio transcripts, and the entity graph, enabling cross-modal, multi-hop reasoning over week-long egocentric videos. The approach achieves state-of-the-art performance on EgoLifeQA and competitive results on Video-MME Long, with substantial gains in tasks requiring relational reasoning across days. By preserving entity-level relationships over time and enabling structured reasoning, EGAgent demonstrates a practical path toward robust longitudinal memory and interpretation in personalized AI systems, while acknowledging limitations in perception accuracy and privacy considerations.

Abstract

The advent of always-on personal AI assistants, enabled by all-day wearable devices such as smart glasses, demands a new level of contextual understanding, one that goes beyond short, isolated events to encompass the continuous, longitudinal stream of egocentric video. Achieving this vision requires advances in long-horizon video understanding, where systems must interpret and recall visual and audio information spanning days or even weeks. Existing methods, including large language models and retrieval-augmented generation, are constrained by limited context windows and lack the ability to perform compositional, multi-hop reasoning over very long video streams. In this work, we address these challenges through EGAgent, an enhanced agentic framework centered on entity scene graphs, which represent people, places, objects, and their relationships over time. Our system equips a planning agent with tools for structured search and reasoning over these graphs, as well as hybrid visual and audio search capabilities, enabling detailed, cross-modal, and temporally coherent reasoning. Experiments on the EgoLifeQA and Video-MME (Long) datasets show that our method achieves state-of-the-art performance on EgoLifeQA (57.5%) and competitive performance on Video-MME (Long) (74.1%) for complex longitudinal video understanding tasks.

Figures (6)

• Figure 1: We show an overview of our EGAgent pipeline for very long video understanding using cross-modal reasoning in ①. Given a very long video and a query, a planning agent devises a multi-step plan of sub-tasks required to answer the query. The planning agent uses a retriever tool to probe three data sources extracted from the long video: audio transcripts, visual frame embeddings, and an entity scene graph, which is the focus of EGAgent. We show an example of how the planning agent composes cross-modal information retrieved from the visual database and entity graph to answer an EgoLife query in ②. We visualize the entity graph query mechanism in ③, where the retriever tool designs a SQL query to retrieve relevant relationships for the planning agent to reason over.
• Figure 2: We use an LLM, denoted as $\mathcal{F}$, to extract an entity graph from text documents $\mathcal{D}$ that represent a very long video, i.e. audio transcripts $\mathcal{AT}$ and scene descriptions and locations extracted from sampled image frames $\mathcal{V}$ (see \ref{['app:implementation']} for details). Each graph relationship $r$ connects a source vertex $v_s$ and target vertex $v_t$ between time $(t_\mathrm{start}, t_\mathrm{end})$. Each vertex has an entity type $\tau(v)$ and the raw text document $d^*$ used to extract the relationship (\ref{['ssec:agentic_framework']}).
• Figure 3: The performance comparison against Gemini 2.5 Pro and EgoButler in each question category in EgoLifeQA. Our approach significantly outperforms baselines on RelationMap (+20.8%) and TaskMaster (+22.2%), where entity understanding and complex reasoning is required to provide a correct answer.
• Figure 5: A walkthrough of our entire EGAgent pipeline (Sec 3.3, main paper) for an example query from EgoLifeQA, with more details in \ref{['app:agent_design']}. At a high-level, given the query, the planning agent comes up with a sequence of 5 sub-tasks, i.e.$S_1$ through $S_5$. Each sub-task is routed to the appropriate search tool $T_i$ followed by the analyzer tool, whose output is appended to the working memory$\mathcal{M} \gets\mathcal{M} \cup \text{Analysis}$. Once all sub-tasks are complete, the original query $Q$ and working memory $\mathcal{M}$ are sent to the VQA agent to predict the answer $A$. The SQL_Query and the details about the entity graph search is illustrated in \ref{['fig:app_full_pipeline_eg_querying']}.
• Figure 6: Here we focus on the entity graph search tool $Tool_\mathrm{eg}$ in the example from \ref{['fig:app_full_pipeline']} and discuss its role in the overall EGAgent pipeline in \ref{['app:agent_design']}. Given the sub-task $S_2$, the planning agent uses a strict-to-relaxed hierarchy to choose a SQL query $q_2$ to search the entity graph to answer the sub-task, i.e. graph entities $\tau(v_s) = \text{Person}$, $r=\text{TALKS\_TO}$ and $(\text{day}, t_{\text{start}},t_{\text{end}})$ to search between. The relevant rows of the SQL table are sent to the analyzer tool, and the relevant inter-entity relationships $(v_s, \tau(v_s), v_t, \tau(v_t), r, t_\mathrm{start}, t_\mathrm{end}, d^*)$ are appended to the working memory$\mathcal{M}$.