ES-MemEval: Benchmarking Conversational Agents on Personalized Long-Term Emotional Support

TL;DR

This work addresses the challenge of maintaining robust long-term memory in personalized emotional-support dialogue systems. It introduces EvoEmo, a multi-session dataset of evolving user states, and ES-MemEval, a three-task benchmark that evaluates five memory capabilities—information extraction, temporal reasoning, conflict detection, abstention, and user modeling—across QA, summarization, and dialogue generation. Through extensive experiments on open-source, commercial, and retrieval-augmented models, the study shows explicit long-term memory is crucial for reducing hallucinations and enabling personalization, while retrieval augmentation improves factual consistency but struggles with temporal dynamics. The findings motivate memory–retrieval integrated designs and adaptive memory strategies to advance reliable, long-term emotional-support dialogue systems.

Abstract

Large Language Models (LLMs) have shown strong potential as conversational agents. Yet, their effectiveness remains limited by deficiencies in robust long-term memory, particularly in complex, long-term web-based services such as online emotional support. However, existing long-term dialogue benchmarks primarily focus on static and explicit fact retrieval, failing to evaluate agents in critical scenarios where user information is dispersed, implicit, and continuously evolving. To address this gap, we introduce ES-MemEval, a comprehensive benchmark that systematically evaluates five core memory capabilities: information extraction, temporal reasoning, conflict detection, abstention, and user modeling, in long-term emotional support settings, covering question answering, summarization, and dialogue generation tasks. To support the benchmark, we also propose EvoEmo, a multi-session dataset for personalized long-term emotional support that captures fragmented, implicit user disclosures and evolving user states. Extensive experiments on open-source long-context, commercial, and retrieval-augmented (RAG) LLMs show that explicit long-term memory is essential for reducing hallucinations and enabling effective personalization. At the same time, RAG improves factual consistency but struggles with temporal dynamics and evolving user states. These findings highlight both the potential and limitations of current paradigms and motivate more robust integration of memory and retrieval for long-term personalized dialogue systems.

Figures (7)

• Figure 1: Excerpt from the EvoEmo dataset, illustrating fragmented disclosures over months. Comprehending why the sister’s engagement evokes overwhelm requires recalling the earlier breakup, emphasizing the importance of robust long-term memory in emotional support dialogues.
• Figure 2: Overview of ES-MemEval, comprising three tasks—QA, summarization, and dialogue generation—designed to evaluate five core capabilities critical for long-term personalized dialogue agents.
• Figure 3: The data generation pipeline of EvoEmo, consisting of three stages: (a) user profile construction, (b) event timeline expansion, and (c) chat data generation, aiming to simulate realistic long-term emotional support conversations.
• Figure 4: Distributions of dialogue topics in EvoEmo and task types in ES-MemEval, covering QA and summarization.
• Figure 5: Distribution of the number of evidence sessions in the QA and summarization tasks of ES-MemEval.