Memento-II: Learning by Stateful Reflective Memory

TL;DR

This paper addresses continual, data-efficient learning for large language model agents by introducing the Stateful Reflective Decision Process (SRDP), which integrates episodic memory with reinforcement learning without updating model parameters. SRDP augments the environment state with an evolving memory, and adopts a two-stage read--write loop where memory-based retrieval guides policy improvement and memory writing provides policy evaluation, yielding a Reflected MDP that supports RL analysis. A Parzen window-based retrieval with KL-regularisation (Parzen-KL) is developed, proving contraction for the KL-regularised evaluation and guaranteeing convergence of the read--write iterations to a fixed point; two-time-scale analysis allows memory updates to evolve on a slower timescale while fast value/policy updates track moving optima. The framework establishes a formal equivalence between memory read/write operations and standard policy iteration, showing that memory-driven reflection can asymptotically realize the optimal policy as memory coverage grows, and provides practical guidance for designing memory-augmented LLM agents with continual adaptation and without gradient updates. Overall, the work offers a principled theoretical foundation for retrieval-based, memory-augmented agents with strong convergence guarantees and clear pathways to real-world continual learning applications.

Abstract

We propose a theoretical framework for continual and experiential learning in large language model agents that integrates episodic memory with reinforcement learning. The framework identifies reflection as the key mechanism that enables agents to adapt through interaction without back propagation or model fine tuning, thereby relaxing the conventional separation between training and deployment.To formalise this process, we introduce the Stateful Reflective Decision Process, which models reflective learning as a two stage read write interaction with episodic memory. Writing stores interaction outcomes and corresponds to policy evaluation, while reading retrieves relevant past cases and corresponds to policy improvement. We show that this process induces an equivalent Markov decision process over augmented state memory representations, allowing the use of classical tools from dynamic programming and reinforcement learning. We further instantiate the framework using entropy regularised policy iteration and establish convergence guarantees. As episodic memory grows and achieves sufficient coverage of the state space, the resulting policy converges to the optimal solution. This work provides a principled foundation for memory augmented and retrieval based language model agents capable of continual adaptation without parameter updates.

Figures (4)

• Figure 1: Learning by Stateful Reflection with Memory realises continual learning without fine-tuning LLMs by iteratively reading from (policy improvement) and writing to (policy evaluation) an evolving episodic memory that drives the agent’s online adaptation. Empirically, this Read-Write Learning paradigm has shown strong effectiveness across diverse domains, including software testingguo2025optimizing, automated data scienceguo2024ds, and deep research agentszhou2025memento, demonstrating that memory-driven reflection can endow LLM agents with genuine self-improving capabilities.
• Figure 2: Three modes of generalised reflection in LLM agents: (a) in-context learning where few-shot exemplars in the prompt steer behaviour on new tasks; (b) post-output reflection via environmental feedback; (c) internal chain-of-thought and planning before emitting an answer.
• Figure 3: A graphical model of the Stateful Reflective Decision Process (SRDP) showing the augmented state space $x_t = (s_t, M_t)$, retrieval actions $c_t$, LLM conditioning, and memory evolution. Dashed boxes indicate augmented states combining environment state and memory.
• Figure 4: The illustration of the assumption of LLM local sufficiency.

Theorems & Definitions (33)

• Definition 1: Reflection
• Definition 2: Stateful Reflective Decision Process (SRDP)
• Definition 3: Reflected MDP
• Lemma 4: also in neu2017unified
• Theorem 7: Convergence of Parzen-KL Soft Policy Iteration
• Theorem 9: Two-Time-Scale Convergence for Parzen-KL Policy Iteration
• Lemma 11: Policy Approximation at State Level
• proof
• Lemma 12: Immediate reward approximation
• proof : Proof