MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

TL;DR

MemoryArena introduces a unified gym for evaluating agent memory in interdependent, multi-session Memory–Action–Environment loops across four domains. By contrasting long-context, external-memory, and retrieval-augmented approaches, the work demonstrates that high performance on traditional memory benchmarks often fails to translate to real, memory-guided agentic tasks, with Group Travel Planning posing the greatest challenge. The results reveal when external memory helps (notably in Progressive Web Search and Formal Reasoning) and why many memory systems underperform in multi-session settings, pointing to representation and training mismatches as key bottlenecks. The study recasts agent memory as a functional component within a memory-driven POMDP-like loop and argues for joint optimization of memory representations and agent training to enable robust, long-horizon decision making in realistic environments.

Abstract

Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.

Figures (19)

• Figure 1: MemoryArena Evaluates agents with Memory with multi-session tasks in a Memory-Agent-Environment Loop.
• Figure 2: MemoryArena supports four distinct evaluation environments, where a memory-augmented task agent completes a sequence of interdependent subtasks. Each subtask session involves multiple agent actions.
• Figure 3: Success Rate at subtask epth $k$. The decay trend indicates agents cannot sustain execution as dependencies span more sessions.
• Figure 4: Data example for bundled web shopping task.
• Figure 5: Data example for the Group Travel Planning task.