Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

Niu Lian, Yuting Wang, Hanshu Yao, Jinpeng Wang, Bin Chen, Yaowei Wang, Min Zhang, Shu-Tao Xia

TL;DR

MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory, is proposed, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization.

Abstract

While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy, which first tries with the abstract Symbolic Schema and progressively "drills down" to the Sensory Buffer and Episodic Stream under high uncertainty. Extensive experiments across 4 benchmarks confirm the effectiveness of MM-Mem on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code is available at https://github.com/EliSpectre/MM-Mem.

From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

TL;DR

MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory, is proposed, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization.

Abstract

While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy, which first tries with the abstract Symbolic Schema and progressively "drills down" to the Sensory Buffer and Episodic Stream under high uncertainty. Extensive experiments across 4 benchmarks confirm the effectiveness of MM-Mem on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code is available at https://github.com/EliSpectre/MM-Mem.
Paper Structure (60 sections, 32 equations, 7 figures, 7 tables)

This paper contains 60 sections, 32 equations, 7 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long Video Understanding
  4. Memory for Agents
  5. Method
  6. Multimodal Pyramid Memory Structure
  7. Sensory Buffer
  8. Episodic Stream
  9. Symbolic Schema
  10. Bottom-Up Memory Construction
  11. Sensory-to-Episodic Memory
  12. Remark: Action-output correspondence
  13. Semantic IB Formulation
  14. A Quality--Quantity Prior
  15. SIB-GRPO: Dynamic Management
  16. ...and 45 more sections

Figures (7)

  • Figure 1: Existing memory paradigms (a-b), inspiration (c), and our insight (d). (a) Vision-centric methods incur redundancy and high latency due to dense visual memories. (b) Text-centric methods suffer from information loss during captioning, leading to hallucination and ambiguity. (c) The natural complementarity between vision and text neatly aligns with the distinction between verbatim and gist traces in Fuzzy-Trace Theory. (d) Our MM-Mem is a bottom-up multimodal memory pyramid from sensory buffer to symbolic schema.
  • Figure 2: Overview of MM-Mem: (left) bottom-up memory formation into sensory, episodic, and symbolic memories; (right) top-down retrieval from schemas to episodic events and sensory details to answer queries.
  • Figure 3: Visualization of ablation results and memory representations.
  • Figure 4: A qualitative example of MM-Mem's coarse-to-fine retrieval across memory layers.
  • Figure 5: HD-EPIC++ is an egocentric long-horizon kitchen video benchmark with highly detailed annotations, covering fine-grained action perception, temporal reasoning, 3D spatial understanding, object motion, gaze, and diverse VQA tasks (e.g., recipes and ingredients).
  • ...and 2 more figures