Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory

Ye Shen, Dun Pei, Yiqiu Guo, Junying Wang, Yijin Guo, Zicheng Zhang, Qi Jia, Jun Zhou, Guangtao Zhai

TL;DR

EvolMem presents a cognitive-driven benchmark for multi-session dialogue memory, covering both declarative and non-declarative memory with fine-grained abilities and a hybrid data synthesis framework that yields diverse, coherent multi-session dialogues. It combines topic-initiated generation and narrative-inspired transformation to create scalable evaluation scenarios with tailored metrics, enabling systematic analysis of memory capabilities in LLMs and agent systems. Results show no single model dominates across all memory dimensions, with pronounced weaknesses in non-declarative memory and efficiency bottlenecks in memory agents, underscoring gaps for practical long-term dialogue applications. The work provides a path toward continual assessment of memory abilities and motivates future improvements in memory strategies for both models and agents.

Abstract

Despite recent advances in understanding and leveraging long-range conversational memory, existing benchmarks still lack systematic evaluation of large language models(LLMs) across diverse memory dimensions, particularly in multi-session settings. In this work, we propose EvolMem, a new benchmark for assessing multi-session memory capabilities of LLMs and agent systems. EvolMem is grounded in cognitive psychology and encompasses both declarative and non-declarative memory, further decomposed into multiple fine-grained abilities. To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations. This framework enables scalable generation of multi-session conversations with controllable complexity, accompanied by sample-specific evaluation guidelines. Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions. Moreover, agent memory mechanisms do not necessarily enhance LLMs' capabilities and often exhibit notable efficiency limitations. Data and code will be released at https://github.com/shenye7436/EvolMem.

EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory

TL;DR

EvolMem presents a cognitive-driven benchmark for multi-session dialogue memory, covering both declarative and non-declarative memory with fine-grained abilities and a hybrid data synthesis framework that yields diverse, coherent multi-session dialogues. It combines topic-initiated generation and narrative-inspired transformation to create scalable evaluation scenarios with tailored metrics, enabling systematic analysis of memory capabilities in LLMs and agent systems. Results show no single model dominates across all memory dimensions, with pronounced weaknesses in non-declarative memory and efficiency bottlenecks in memory agents, underscoring gaps for practical long-term dialogue applications. The work provides a path toward continual assessment of memory abilities and motivates future improvements in memory strategies for both models and agents.

Abstract

Despite recent advances in understanding and leveraging long-range conversational memory, existing benchmarks still lack systematic evaluation of large language models(LLMs) across diverse memory dimensions, particularly in multi-session settings. In this work, we propose EvolMem, a new benchmark for assessing multi-session memory capabilities of LLMs and agent systems. EvolMem is grounded in cognitive psychology and encompasses both declarative and non-declarative memory, further decomposed into multiple fine-grained abilities. To construct the benchmark, we introduce a hybrid data synthesis framework that consists of topic-initiated generation and narrative-inspired transformations. This framework enables scalable generation of multi-session conversations with controllable complexity, accompanied by sample-specific evaluation guidelines. Extensive evaluation reveals that no LLM consistently outperforms others across all memory dimensions. Moreover, agent memory mechanisms do not necessarily enhance LLMs' capabilities and often exhibit notable efficiency limitations. Data and code will be released at https://github.com/shenye7436/EvolMem.
Paper Structure (34 sections, 4 equations, 7 figures, 8 tables, 1 algorithm)

This paper contains 34 sections, 4 equations, 7 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Evolution of LLMs & Agents
  4. Memory Evaluation Benchmarks
  5. EvolMem
  6. Memory Classification
  7. Dialogue synthesis framework
  8. Topic-Initiated Generation
  9. Narrative-Inspired Transformation
  10. Evaluation Methods
  11. Experiment Setup
  12. Data Construction and Statistics
  13. LLMs For Evaluation
  14. Agent Memory Systems For Evaluation
  15. Results and Analysis
  16. ...and 19 more sections

Figures (7)

  • Figure 1: The leaderboard and capability distribution of LLMs across fine-grained memory dimensions.
  • Figure 2: The outline of our hybrid data synthesis framework and adopted evaluation metrics.
  • Figure 3: The Spearman's Correlation between the rankings under different generators and overall rankings.
  • Figure 4: The performance of DeepSeek-V3.2 under different number of sessions.
  • Figure 5: Spearman's Correlation between EvolMem and other four leaderboards.
  • ...and 2 more figures