Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EverMemBench: Benchmarking Long-Term Interactive Memory in Large Language ModelsEverMemBench: Benchmarking Long-Term Interactive Memory in Large Language Models

Chuanrui Hu, Tong Li, Xingze Gao, Hongda Chen, Dannong Xu, Yi Bai, Tianwei Lin, Xinda Zhao, Xiaohong Li, Jiaqi An, Yunyun Han, Jian Pei, Yafeng Deng

TL;DR

EverMemBench tackles the challenge of long-term conversational memory in large language models by constructing a high-fidelity, multi-party benchmark with 1 million tokens across five projects and three memory-evaluation dimensions: fine-grained recall, memory awareness, and user profile understanding. It deploys a three-stage data pipeline—blueprint generation, dialogue synthesis with hierarchical summaries, and evidence-grounded QA—to stress attribution, temporal tracking, and dynamic knowledge updates in realistic social contexts. The results reveal that multi-hop reasoning collapses in multi-party interleaved settings, temporal reasoning requires version semantics beyond simple timestamps, and retrieval-based memory remains a critical bottleneck for strong models, despite near-perfect performance when full context is available. EverMemBench thereby provides a diagnostic platform that highlights the interaction between retrieval quality and reasoning, guiding the design of next-generation memory architectures for robust, socially aware long-horizon agents in real-world collaboration.

Abstract

Long-term conversational memory is essential for LLM-based assistants, yet existing benchmarks focus on dyadic, single-topic dialogues that fail to capture real-world complexity. We introduce EverMemBench, a benchmark featuring multi-party, multi-group conversations spanning over 1 million tokens with temporally evolving information, cross-topic interleaving, and role-specific personas. EverMemBench evaluates memory systems across three dimensions through 1,000+ QA pairs: fine-grained recall, memory awareness, and user profile understanding. Our evaluation reveals critical limitations: (1) multi-hop reasoning collapses in multi-party settings, with even oracle models achieving only 26%; (2) temporal reasoning remains unsolved, requiring version semantics beyond timestamp matching; (3) memory awareness is bottlenecked by retrieval, where current similarity-based methods fail to bridge the semantic gap between queries and implicitly relevant memories. EverMemBench provides a challenging testbed for developing next-generation memory architectures.

EverMemBench: Benchmarking Long-Term Interactive Memory in Large Language ModelsEverMemBench: Benchmarking Long-Term Interactive Memory in Large Language Models

TL;DR

EverMemBench tackles the challenge of long-term conversational memory in large language models by constructing a high-fidelity, multi-party benchmark with 1 million tokens across five projects and three memory-evaluation dimensions: fine-grained recall, memory awareness, and user profile understanding. It deploys a three-stage data pipeline—blueprint generation, dialogue synthesis with hierarchical summaries, and evidence-grounded QA—to stress attribution, temporal tracking, and dynamic knowledge updates in realistic social contexts. The results reveal that multi-hop reasoning collapses in multi-party interleaved settings, temporal reasoning requires version semantics beyond simple timestamps, and retrieval-based memory remains a critical bottleneck for strong models, despite near-perfect performance when full context is available. EverMemBench thereby provides a diagnostic platform that highlights the interaction between retrieval quality and reasoning, guiding the design of next-generation memory architectures for robust, socially aware long-horizon agents in real-world collaboration.

Abstract

Long-term conversational memory is essential for LLM-based assistants, yet existing benchmarks focus on dyadic, single-topic dialogues that fail to capture real-world complexity. We introduce EverMemBench, a benchmark featuring multi-party, multi-group conversations spanning over 1 million tokens with temporally evolving information, cross-topic interleaving, and role-specific personas. EverMemBench evaluates memory systems across three dimensions through 1,000+ QA pairs: fine-grained recall, memory awareness, and user profile understanding. Our evaluation reveals critical limitations: (1) multi-hop reasoning collapses in multi-party settings, with even oracle models achieving only 26%; (2) temporal reasoning remains unsolved, requiring version semantics beyond timestamp matching; (3) memory awareness is bottlenecked by retrieval, where current similarity-based methods fail to bridge the semantic gap between queries and implicitly relevant memories. EverMemBench provides a challenging testbed for developing next-generation memory architectures.
Paper Structure (35 sections, 5 equations, 1 figure, 4 tables)

This paper contains 35 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. EverMemBench
  4. Evaluation Dimensions
  5. Task Formulation
  6. Data Construction
  7. Preliminaries
  8. Blueprint Generation
  9. Conversation Generation
  10. Q&A Generation
  11. Fine-grained Recall
  12. Memory Awareness
  13. Profile Understanding
  14. Q&A Quality Control
  15. Phase I: Blind Test
  16. ...and 20 more sections

Figures (1)

  • Figure 1: Data curation pipeline of EverMemBench.Stage 1 instantiates a synthetic organization and generates agent profiles, including stable attributes (e.g., roles and hard skills), context-dependent communication styles, and role-consistent sub-tasks. Stage 2 constructs a globally consistent, calendar-aligned timeline from project plans and dated public news, and incrementally synthesizes long-horizon group-chat dialogue blocks conditioned on profiles and rolling summaries; each block is iteratively validated and repaired via programmatic and semantic checks. Stage 3 generates evidence-grounded QA items targeting fine-grained recall, memory awareness, and user-profile understanding, and filters them through a three-phase quality control procedure: LLM blind testing, answerability/uniqueness verification, and human audit.