VideoLucy: Deep Memory Backtracking for Long Video Understanding

TL;DR

VideoLucy addresses the core difficulty of long-video understanding: capturing temporal context and preserving fine-grained details under costly dense captioning. It introduces a deep memory backtracking framework with a hierarchical memory that progresses from coarse to fine representations and a set of agent-guided loops that iteratively retrieve question-relevant memories. The EgoMem benchmark assesses both cross-temporal event understanding and instantaneous detail perception in ultra-long videos. Empirical results on multiple benchmarks show VideoLucy achieving state-of-the-art or competitive performance using open-source models, highlighting the practical value of memory-guided reasoning for long-form video analysis.

Abstract

Recent studies have shown that agent-based systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as a promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the cost of dense frame-level captioning, they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limitations, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, VideoLucy employs a hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. Through an agent-based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information is gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding. EgoMem is designed to comprehensively evaluate a model's ability to understand complex events that unfold over time and capture fine-grained details in extremely long videos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, VideoLucy significantly outperforms state-of-the-art methods on multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly at https://videolucy.github.io

Figures (10)

• Figure 1: Comparison between our VideoLucy with existing video agent-based systems ma2024drvideowang2024videoagent. In (a), they usually perform frame-level captioning on sparsely sampled frames, and then search for information, resulting in great information loss and hampering temporal understanding. In (b), our VideoLucy, through a hierarchical memory structure and a memory backtracking mechanism, effectively performs multi-level video representation and achieves comprehensive information coverage.
• Figure 2: A toy example of hierarchical memory structure: For a video clip of a certain time segment, as the memory hierarchy deepens, the number of frames captured per second increases, and the time span covered by the memory shortens, thereby achieving multi-level video representations with comprehensive information coverage.
• Figure 3: Question types and examples in the EgoMem benchmark. We design six question types to comprehensively assess the model's understanding of cross-temporal events in long videos. Additionally, a detail-perception task evaluates its ability to capture instantaneous visual features. All the questions are annotated manually, accompanied by sufficient evidence descriptions.
• Figure 4: Results on the "Needle-in-A-Video-Haystack" evaluation with 10 long videos.
• Figure 5: Evaluation on information richness and relevance in memory backtracking of VideoLucy.