Towards Multimodal Lifelong Understanding: A Dataset and Agentic Baseline

TL;DR

The Recursive Multimodal Agent (ReMA) is proposed, which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods.

Abstract

While datasets for video understanding have scaled to hour-long durations, they typically consist of densely concatenated clips that differ from natural, unscripted daily life. To bridge this gap, we introduce MM-Lifelong, a dataset designed for Multimodal Lifelong Understanding. Comprising 181.1 hours of footage, it is structured across Day, Week, and Month scales to capture varying temporal densities. Extensive evaluations reveal two critical failure modes in current paradigms: end-to-end MLLMs suffer from a Working Memory Bottleneck due to context saturation, while representative agentic baselines experience Global Localization Collapse when navigating sparse, month-long timelines. To address this, we propose the Recursive Multimodal Agent (ReMA), which employs dynamic memory management to iteratively update a recursive belief state, significantly outperforming existing methods. Finally, we establish dataset splits designed to isolate temporal and domain biases, providing a rigorous foundation for future research in supervised learning and out-of-distribution generalization.

Figures (8)

• Figure 1: Physical Temporal Span vs. Scale. The x-axis represents the Physical Temporal Span ($T_{span}$), while bubble size indicates Observational Duration ($T_{dur}$). Unlike existing datasets clustered in the bottom-left (short clips, $T_{span} \approx T_{dur}$), MM-Lifelong occupies the unique Lifelong Regime (top-right). This regime is characterized by high temporal sparsity ($T_{span} \gg T_{dur}$), requiring models to bridge unobserved gaps across days to months.
• Figure 2: Performance Scaling Analysis. As the number of input frames increases, end-to-end MLLMs initially improve but soon exhibit performance oscillation and even sharp degradation due to context saturation and noise accumulation. In contrast, ReMA consistently scales with more recursion rounds, effectively mitigating this bottleneck via dynamic memory management and demonstrating superior scaling potential and stability.
• Figure 3: 1) Live stream subset of MM-Lifelong comprises 105.6 hours of broadcast footage spanning 51 days. 2) An example of a multi-clue (hop) reasoning question with an ultra-long temporal certificate: The task requires identifying all occurrences where the streamer sings a specific song on subways across multiple cities. Successfully answering this requires persistent memory and the ability to perform multi-event inference over more than 10 hours of continuous livestream data.
• Figure 4: Distribution of question categories.
• Figure 5: Distribution of video clip domains.