ProcMEM: Learning Reusable Procedural Memory from Experience via Non-Parametric PPO for LLM Agents

TL;DR

ProcMEM tackles the inefficiency of re-deriving solutions in recurring tasks by learning reusable procedural memory for LLM agents. It introduces a Skill-MDP formalism to organize decision-making around executable Skills and employs Non-Parametric PPO to evolve the Skill pool without updating the LLM, using Semantic Gradients and a PPO-style PPO Gate for safe verification. Key contributions include a structured Skill representation, a non-parametric optimization loop, and online score-based maintenance that yields high reuse and extreme memory compression. Empirical results on TextArena and ALFWorld demonstrate superior reuse rates, enhanced task performance, and substantial memory efficiency, supporting long-horizon, autonomous operation of LLM-driven agents.

Abstract

LLM-driven agents demonstrate strong performance in sequential decision-making but often rely on on-the-fly reasoning, re-deriving solutions even in recurring scenarios. This insufficient experience reuse leads to computational redundancy and execution instability. To bridge this gap, we propose ProcMEM, a framework that enables agents to autonomously learn procedural memory from interaction experiences without parameter updates. By formalizing a Skill-MDP, ProcMEM transforms passive episodic narratives into executable Skills defined by activation, execution, and termination conditions to ensure executability. To achieve reliable reusability without capability degradation, we introduce Non-Parametric PPO, which leverages semantic gradients for high-quality candidate generation and a PPO Gate for robust Skill verification. Through score-based maintenance, ProcMEM sustains compact, high-quality procedural memory. Experimental results across in-domain, cross-task, and cross-agent scenarios demonstrate that ProcMEM achieves superior reuse rates and significant performance gains with extreme memory compression. Visualized evolutionary trajectories and Skill distributions further reveal how ProcMEM transparently accumulates, refines, and reuses procedural knowledge to facilitate long-term autonomy.

Figures (6)

• Figure 1: Episodic memory versus procedural memory in LLM-driven agents. Episodic memory retrieves past interactions for reference, requiring inference-heavy reasoning at decision time. Procedural memory encodes reusable executable Skills that directly map situations to actions, enabling efficient experience reuse.
• Figure 2: Overview of the ProcMEM framework.(Left) Skill-MDP: The agent selects a Skill $\omega$ based on state $s_t$ and activation conditions. A frozen LLM executes $\omega$ into primitive actions $a_t$ over multiple steps until termination. Post-episode trajectories $\mathcal{T}$ are stored in a buffer. (Middle) Procedural Memory: Skills are dynamically managed via refinement, generation, and score-based pruning to maintain pool quality. (Right) Non-Parametric PPO: Evolution proceeds in two stages: ① Semantic Gradient: Derives and aggregates per-trajectory gradients through hindsight attribution to generate candidates $\omega'$. ② PPO Gate: Filters candidates via trust-region verification, admitting only the best-performing valid candidate into the Skill pool.
• Figure 3: Training curves of ProcMEM and ablation variants. Solid lines and shaded areas denote the smoothed mean and standard deviation of average returns, respectively.
• Figure 4: Evolutionary Lineage of Skills. Gray bars represent Skill lifespans along the evolutionary timeline (horizontal axis). Dashed vertical lines denote refinement events where a parent Skill evolves into children; multiple lines indicate repeated refinements. Red 'X' markers signify pruning of underperforming variants for pool efficiency. The dark blue arrow and sequential alignment (e.g., $v_1 \to v_{13}$) track the sustained trajectory of Skill evolution.
• Figure 5: Skill distribution across LLM agents and task complexities. Bars represent the empirical invocation probability for each skill categorized by different LLM backbones (top) and task difficulty levels in Mastermind-v0 (bottom). $N$ denotes the average number of skill invocations per episode.