A Benchmark for Procedural Memory Retrieval in Language Agents

TL;DR

This work tackles the challenge of procedural memory retrieval in language agents by isolating retrieval from execution. It introduces a benchmark built on ALFWorld with dual corpora (expert and LLM-generated trajectories) and a coverage-balanced query set to rigorously test cross-context generalization. The key finding is a generalization cliff: embedding-based retrievers excel in familiar vocabularies but fail under vocabulary shifts, whereas LLM-generated procedural abstractions transfer more reliably. The study provides a diagnostic framework, design guidance for retrieval systems, and open-source resources to advance dependable cross-context procedural transfer.

Abstract

Current AI agents excel in familiar settings, but fail sharply when faced with novel tasks with unseen vocabularies -- a core limitation of procedural memory systems. We present the first benchmark that isolates procedural memory retrieval from task execution, evaluating whether agents can recognize functionally equivalent procedures that span different object instantiations. Using ALFWorld, we construct dual corpora of expert and LLM-generated trajectories and evaluate six retrieval methods using systematically stratified queries. Our results expose a clear generalization cliff: embedding-based methods perform strongly on familiar contexts, yet degrade considerably on novel ones, while LLM-generated procedural abstractions demonstrate reliable cross-context transfer. Controlled ablations show that although embeddings capture some lexical-level abstraction, they fundamentally treat procedures as unordered bags of words, discarding temporal structure necessary for cross-context transfer. Corpus scale delivers far larger gains than representation enrichment, revealing an architectural ceiling in current encoders. Our benchmark offers the first diagnostic framework separating genuine procedural understanding from surface-level memorization and gives tools for developing retrieval systems capable of dependable generalization. Resources available at our GitHub repository (https://github.com/qpiai/Proced_mem_bench).

Figures (4)

• Figure 1: Overview of the procedural retrieval problem and benchmark pipeline. Left: Embedding-based retrieval confuses transformation types under vocabulary shift (e.g., CLEAN vs. HEAT vs. COOL), producing structurally incorrect matches. Right: Our benchmark pipeline evaluates retrieval quality via action-only and state-aware representations, LLM-judged relevance scoring, and standardized IR metrics.
• Figure 2: Threshold sensitivity analysis documenting MAP performance and percentage of zero-relevant queries for thresholds 6-10. Threshold $\geq 6$ delivers the best balance between retrieval quality and corpus coverage.
• Figure 3: t-SNE visualization of 327 ALFWorld tasks. Seen and unseen tasks display weak separation (Cohen's $d\approx 0.02$-$0.04$), pointing to considerable semantic overlap.
• Figure 4: Human-LLM agreement on procedural relevance judgments (binary threshold at 6/10). High specificity confirms reliable failure detection, while disagreements arise primarily from partial procedural utility recognized by humans.

Theorems & Definitions (4)

• Definition 1: Procedural Trajectory
• Definition 2: Procedural Similarity
• Definition 3: Cross-Context Retrieval Problem
• Definition 4: Generalization Gap