Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents

Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, Hendrik Hamann, Jingrui He, Hanghang Tong

TL;DR

Mem-Gallery tackles the challenge of evaluating multimodal long-term conversational memory in MLLM agents by introducing a large, multi-session dataset grounded in both images and text, and a three-dimensional evaluation framework spanning memory extraction/adaptation, memory reasoning, and memory knowledge management. The benchmark reveals that explicit multimodal information retention and principled memory organization are crucial, yet current memory systems struggle with reasoning over evolving multimodal evidence and updating knowledge in the presence of conflicts. Through benchmarking thirteen memory mechanisms across diverse backbones, the work highlights efficiency bottlenecks and the importance of selective, high-quality retrieval over sheer memory volume. The Mem-Gallery framework provides a practical, scalable platform to guide future research in robust, long-horizon multimodal memory for real-world MLLM agents.

Abstract

Long-term memory is a critical capability for multimodal large language model (MLLM) agents, particularly in conversational settings where information accumulates and evolves over time. However, existing benchmarks either evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts, failing to evaluate how multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich multimodal dependencies. Building on this dataset, we propose a systematic evaluation framework that assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. Extensive benchmarking across thirteen memory systems reveals several key findings, highlighting the necessity of explicit multimodal information retention and memory organization, the persistent limitations in memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.

Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents

TL;DR

Mem-Gallery tackles the challenge of evaluating multimodal long-term conversational memory in MLLM agents by introducing a large, multi-session dataset grounded in both images and text, and a three-dimensional evaluation framework spanning memory extraction/adaptation, memory reasoning, and memory knowledge management. The benchmark reveals that explicit multimodal information retention and principled memory organization are crucial, yet current memory systems struggle with reasoning over evolving multimodal evidence and updating knowledge in the presence of conflicts. Through benchmarking thirteen memory mechanisms across diverse backbones, the work highlights efficiency bottlenecks and the importance of selective, high-quality retrieval over sheer memory volume. The Mem-Gallery framework provides a practical, scalable platform to guide future research in robust, long-horizon multimodal memory for real-world MLLM agents.

Abstract

Long-term memory is a critical capability for multimodal large language model (MLLM) agents, particularly in conversational settings where information accumulates and evolves over time. However, existing benchmarks either evaluate multi-session memory in text-only conversations or assess multimodal understanding within localized contexts, failing to evaluate how multimodal memory is preserved, organized, and evolved across long-term conversational trajectories. Thus, we introduce Mem-Gallery, a new benchmark for evaluating multimodal long-term conversational memory in MLLM agents. Mem-Gallery features high-quality multi-session conversations grounded in both visual and textual information, with long interaction horizons and rich multimodal dependencies. Building on this dataset, we propose a systematic evaluation framework that assesses key memory capabilities along three functional dimensions: memory extraction and test-time adaptation, memory reasoning, and memory knowledge management. Extensive benchmarking across thirteen memory systems reveals several key findings, highlighting the necessity of explicit multimodal information retention and memory organization, the persistent limitations in memory reasoning and knowledge management, as well as the efficiency bottleneck of current models.
Paper Structure (58 sections, 11 equations, 18 figures, 14 tables)

This paper contains 58 sections, 11 equations, 18 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-Round Conversational Benchmark
  4. Long-Term Agent Memory
  5. Mem-Gallery Benchmark
  6. Benchmark Construction
  7. Conversation Data
  8. Evaluation Data
  9. Conversational Environment for Memory
  10. Evaluation Framework
  11. Memory Extraction & Adaptation
  12. Memory Reasoning
  13. Memory Knowledge Management
  14. Benchmarking Analysis
  15. Evaluation Setup
  16. ...and 43 more sections

Figures (18)

  • Figure 1: The conceptual illustration of Mem-Gallery.
  • Figure 2: Effectiveness analysis of visual information with representative memory models. Compared with LoCoMo maharana2024evaluating, visual information plays a much more critical role in Mem-Gallery.
  • Figure 3: The overall pipeline of our proposed Mem-Gallery.
  • Figure 4: Taxonomy and distribution of evaluation tasks.
  • Figure 5: Efficiency comparison results on information memorization time (top) and memory retrieval & answer generation time (bottom) in seconds with the log scale.
  • ...and 13 more figures