RealMem: Benchmarking LLMs in Real-World Memory-Driven Interaction

TL;DR

RealMem addresses the gap in memory evaluation for long-term, project-oriented interactions by introducing a large-scale, realistic benchmark and a three-stage synthesis pipeline that evolves memory across interleaved sessions and multiple projects. The approach combines Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate dynamic memory states; it evaluates multiple memory architectures (e.g., Mem0, A-mem, MemoryOS, Graph Memory) using retrieval and semantic metrics plus LLM-based judgments. Experiments reveal substantial challenges for current memory systems in maintaining coherent project threads, with domain-dependent performance and a notable gap to Oracle upper bounds, underscoring the need for improved memory integration in real-world autonomous agents. RealMem provides a rigorous, scalable platform for diagnosing and guiding advances in long-term memory, memory scheduling, and proactive alignment in AI agents that must operate across evolving goals and tasks.

Abstract

As Large Language Models (LLMs) evolve from static dialogue interfaces to autonomous general agents, effective memory is paramount to ensuring long-term consistency. However, existing benchmarks primarily focus on casual conversation or task-oriented dialogue, failing to capture **"long-term project-oriented"** interactions where agents must track evolving goals. To bridge this gap, we introduce **RealMem**, the first benchmark grounded in realistic project scenarios. RealMem comprises over 2,000 cross-session dialogues across eleven scenarios, utilizing natural user queries for evaluation. We propose a synthesis pipeline that integrates Project Foundation Construction, Multi-Agent Dialogue Generation, and Memory and Schedule Management to simulate the dynamic evolution of memory. Experiments reveal that current memory systems face significant challenges in managing the long-term project states and dynamic context dependencies inherent in real-world projects. Our code and datasets are available at [https://github.com/AvatarMemory/RealMemBench](https://github.com/AvatarMemory/RealMemBench).

Figures (6)

• Figure 1: Comparison of three interaction paradigms in human--agent interactions: (a) casual conversation, (b) task-oriented dialogue, and (c) long-term project-oriented interactions spanning multiple sessions with interleaved projects and evolving context.
• Figure 2: Examples of four query types in RealMem: (1) Temporal Reasoning resolves temporal constraints and schedule conflicts; (2) Static Retrieval ensures continuity by recalling accumulated context; (3) Dynamic Updating synchronizes memory with evolving project states; and (4) Proactive Alignment leverages user memory to anticipate implicit intents and goals.
• Figure 3: Overview of the data synthesis framework. The pipeline consists of three cascaded stages: (1) Project Foundation Construction, which initializes user personas and hierarchical project skeletons (i.e., blueprints, events, and sessions); (2) Multi-Agent Dialogue Generation, where the User Agent and Assistant Agent simulate interactions based on the session queue and dynamic context; and (3) Memory and Schedule Management, which iteratively retrieves, updates, and deduplicates memory points and schedule tables to ensure long-term consistency.
• Figure 4: Average performance scores of MemoryOS across various topics.
• Figure 5: The specific prompt used for evaluating the consistency between the generated response and user memory.