MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory

TL;DR

MemRL addresses how to empower agents to self-evolve without fine-tuning the backbone model. It reframes retrieval as a value-based decision over an external Intent-Experience-Utility memory and couples a Two-Phase Retrieval with non-parametric RL to update memory utilities, yielding continuous improvement while preserving stability. The authors prove convergence properties and bounded variance, and demonstrate superior performance and generalization across four diverse benchmarks, validating MemRL as an effective trajectory verifier that filters brittle policies. This approach offers a practical paradigm for deploying self-improving agents that adapt through interaction without catastrophic forgetting or expensive parameter updates.

Abstract

The hallmark of human intelligence is the ability to master new skills through Constructive Episodic Simulation-retrieving past experiences to synthesize solutions for novel tasks. While Large Language Models possess strong reasoning capabilities, they struggle to emulate this self-evolution: fine-tuning is computationally expensive and prone to catastrophic forgetting, while existing memory-based methods rely on passive semantic matching that often retrieves noise. To address these challenges, we propose MemRL, a framework that enables agents to self-evolve via non-parametric reinforcement learning on episodic memory. MemRL explicitly separates the stable reasoning of a frozen LLM from the plastic, evolving memory. Unlike traditional methods, MemRL employs a Two-Phase Retrieval mechanism that filters candidates by semantic relevance and then selects them based on learned Q-values (utility). These utilities are continuously refined via environmental feedback in an trial-and-error manner, allowing the agent to distinguish high-value strategies from similar noise. Extensive experiments on HLE, BigCodeBench, ALFWorld, and Lifelong Agent Bench demonstrate that MemRL significantly outperforms state-of-the-art baselines. Our analysis experiments confirm that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates.

Figures (11)

• Figure 1: Benchmark Runtime Learning performance of MemRL. We compare MemRL against state-of-the-art memory baselines (MemP) and standard retrieval methods (RAG). MemRL consistently outperforms various baselines, demonstrating the efficacy of runtime utility-driven updates.
• Figure 2: The conceptual framework of MemRL.
• Figure 3: An illustrative example of memory-augmented decision making under a Markov Decision Process. At time step $t$, the agent starts with an initial memory set $\mathcal{M}_t$. At time step $t{+}1$, an intent (Intent A) retrieves relevant past experiences, but initially leads to a failed generation. In contrast, another intent (Intent B) succeeds and its associated experience is added to memory. At time step $t{+}2$, Intent A retrieves the newly stored successful experience from Intent B, resulting in a successful outcome. This example shows how memory retrieval enables knowledge reuse across intents, implicitly supporting transfer across tasks through shared experiences.
• Figure 4: Overview of the MemRL Framework.(Top)The end-to-end learning loop: given a query $\mathbf{s}$, the agent retrieves context $\mathbf{m}_{ctx}$ from memory $\mathbf{M}$, generates output $\mathbf{y}$, and updates memory value $Q$ based on reward $R$. (Bottom Left)Two-Phase Retrieval: Candidates are recalled via similarity, then re-ranked using learned Q-values. (Bottom Right)Utility Update: Memory values ($Q$) are updated using environmental rewards to distinguish functional utility from semantic similarity.
• Figure 5: OS Interaction Performance. (a) MemRL demonstrates superior stability and higher peak performance compared to the baseline. (b) The widening gap in CSR illustrates that RL-enhanced methods effectively accumulate solved tasks over time.

Theorems & Definitions (1)

• Theorem 1