EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents

TL;DR

EMemBench introduces an interactive, trajectory-conditioned benchmark that evaluates episodic memory in VLM agents by converting each agent’s episode into a ground-truth QA set drawn from its own experiences. The framework supports both text-only and vision-language environments (Jericho and Crafter), using a programmable generator, structured logging, and robust ground-truth computation to enable automated, individualized, and reproducible memory assessment. Results reveal persistent bottlenecks in induction and spatial reasoning, especially in visually grounded settings, and show memory-augmented agents provide clear gains on text tasks but more limited improvements on visual tasks. A human study confirms the difficulty of EMemBench, and analyses highlight the importance of interactive evaluation, horizon control, and cross-benchmark comparisons for diagnosing memory capabilities in modern agents.

Abstract

We introduce EMemBench, a programmatic benchmark for evaluating long-term memory of agents through interactive games. Rather than using a fixed set of questions, EMemBench generates questions from each agent's own trajectory, covering both text and visual game environments. Each template computes verifiable ground truth from underlying game signals, with controlled answerability and balanced coverage over memory skills: single/multi-hop recall, induction, temporal, spatial, logical, and adversarial. We evaluate memory agents with strong LMs/VLMs as backbones, using in-context prompting as baselines. Across 15 text games and multiple visual seeds, results are far from saturated: induction and spatial reasoning are persistent bottlenecks, especially in visual setting. Persistent memory yields clear gains for open backbones on text games, but improvements are less consistent for VLM agents, suggesting that visually grounded episodic memory remains an open challenge. A human study further confirms the difficulty of EMemBench.

Figures (11)

• Figure 1: EMemBench overview. An agent interacts with game environment to produce an episode trajectory. We log agent-observable signals and all underlying game signals. A carefully designed algorithm converts each episode into a QA set with calculated ground truths, and the same agent then answers these questions using only agent-observable context plus its own memory.
• Figure 2: Performance gap between interactive evaluation and a fixed QA setting for GPT-5.1 on text-only games. Each bar reports $\Delta\mathrm{acc}=\mathrm{acc}_{\text{interactive}}-\mathrm{acc}_{\text{fixed}}$ (percentage points) for a question category; the last two columns show Overall Acc and Overall F1. Positive values indicate interactive evaluation is more favorable ("interactive higher"), while negative values indicate the fixed setting is more favorable ("FIXED higher").
• Figure 3: Performance vs. rounds of play. Each dot is one run, and each color is one method.
• Figure 4: Query horizon control under in-context baseline. We restrict evidence and questions to steps 1--50. Each radar compares the standard setting vs. the horizon-controlled setting across seven skill categories.
• Figure 5: UI interface of the application for human evaluation question answering.

Theorems & Definitions (2)

• Definition 1: Interaction episode
• Definition 2: Benchmark generator