Episodic Memories Generation and Evaluation Benchmark for Large Language Models

TL;DR

The paper defines episodic memory as time- and space-grounded recall and argues that current LLMs lack robust episodic memory, which causes confabulations and poor long-range reasoning. It introduces a cognitive-science–inspired framework and a contamination-free synthetic benchmark to model and evaluate episodic memory in LLMs, including an open-source framework and 11 datasets. By evaluating GPT-4o, Claude variants, Llama 3.1, and o1-mini across in-context, retrieval-augmented, and fine-tuning memory setups, the study finds that even top models struggle with tasks requiring tracking multiple related events and complex spatio-temporal relations, and that naive fine-tuning does not generalize beyond single-event memorization. These results underscore the need for new architectures and training paradigms to enable persistent, grounded episodic memory in AI systems, with the benchmark serving as a scalable, controllable testbed for future work.

Abstract

Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.

Figures (8)

• Figure 1: Memory recall process implemented in the benchmark: a set of events match a given cue, from which a list of elements are recalled (here, all entities that have been seen in a given space).
• Figure 2: Book generation: skewed event sampling (Appendix \ref{['app:1book:events_generation']}), LLM-based chapter generation with quality control (Appendix \ref{['sec:verif_direct']}, \ref{['sec:verif_llm']}), and chapter concatenation.
• Figure 3: Overall performance comparison: Critical distance plot ranking all LLM models and memory combinations (instances not tied by an horizontal bar are statistically different).
• Figure 4: Impact of cue type. F1-score across different cue types (y-axis) for models ordered according to their overall rank in Fig. \ref{['fig:cd200']} (x-axis) and for increasing number of events that match the cue (from left to right sub-plots).
• Figure 5: Position of the event features (time, space, entity, content details) relative to the book (top), the chapter (middle), the paragraph (bottom), for Claude 3.5 Sonnet (left) and GPT-4o (right), with $N_{\text{events}} = 200$.