PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?

TL;DR

PersistBench introduces a targeted benchmark to diagnose safety risks from long-term memories in LLMs, focusing on cross-domain leakage and memory-induced sycophancy while also measuring beneficial memory use. The authors construct 500 high-quality, human-validated memory–query samples via Monte Carlo Tree Search, evaluated across 18 frontier and open-weight LLMs with a judge-based scoring framework and a bootstrap-based uncertainty estimation. Key findings reveal a median cross-domain leakage failure rate around 53% and a sycophancy failure rate near 98%, with beneficial memory use showing weaker but nontrivial failure rates. The study also investigates defenses, showing that carefully designed prompts (e.g., GEPA-inspired) can trade off leakage and sycophancy more efficiently than naive baselines, highlighting that safety requires more than task-level prompting and may require memory curation and task-aware memory usage. Overall, PersistBench provides a rigorous, extensible framework for evaluating when LLMs should remember or forget to balance personalization with safety and reliability.

Abstract

Conversational assistants are increasingly integrating long-term memory with large language models (LLMs). This persistence of memories, e.g., the user is vegetarian, can enhance personalization in future conversations. However, the same persistence can also introduce safety risks that have been largely overlooked. Hence, we introduce PersistBench to measure the extent of these safety risks. We identify two long-term memory-specific risks: cross-domain leakage, where LLMs inappropriately inject context from the long-term memories; and memory-induced sycophancy, where stored long-term memories insidiously reinforce user biases. We evaluate 18 frontier and open-source LLMs on our benchmark. Our results reveal a surprisingly high failure rate across these LLMs - a median failure rate of 53% on cross-domain samples and 97% on sycophancy samples. To address this, our benchmark encourages the development of more robust and safer long-term memory usage in frontier conversational systems.

Figures (28)

• Figure 1: Persistent long-term memory is reused during inference in conversational assistants. While such memory enables personalization, it can also lead to cross-domain leakage and memory-induced sycophancy, which are evaluated in PersistBench.
• Figure 2: PersistBench generation pipeline. Candidate memory–query pairs are generated to target specific failure modes, validated against held-out models, and finally reviewed by human annotators for quality and realism.
• Figure 3: Reasoning vs Non-Reasoning: Qwen3; Kimi-K2
• Figure 4: Model Size Comparison: Llama-3.1-8B vs Llama-3.3-70B; GPT-OSS-20B vs GPT-OSS-120B
• Figure 5: Defensive Prompt Pareto Plot (avg. across LLMs)