EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems

TL;DR

This work introduces the Jericho Test-Time Learning (J-TTL) benchmark to quantify an agent's ability to improve within a single test session and proposes EvoTest, a gradient-free, whole-system evolutionary framework that between episodes rewrites the agent's configuration (prompt, memory, hyperparameters, and tool-use routines) based on full episode transcripts. By separating acting from adaptation and employing a UCB-driven selection of configurations, EvoTest achieves superior data efficiency and consistently outperforms memory-, reflection-, prompt-optimization-, and online-fine-tuning baselines across six Jericho games, even winning two games. The approach leverages rich, narrative feedback for credit assignment, offering a practical and scalable path toward self-improving agentic systems that function well under sparse reward conditions. The work thereby advances the design of autonomous agents capable of rapid test-time learning and self-optimization in complex, language-rich environments.

Abstract

A fundamental limitation of current AI agents is their inability to learn complex skills on the fly at test time, often behaving like "clever but clueless interns" in novel environments. This severely limits their practical utility. To systematically measure and drive progress on this challenge, we first introduce the Jericho Test-Time Learning (J-TTL) benchmark. J-TTL is a new evaluation setup where an agent must play the same game for several consecutive episodes, attempting to improve its performance from one episode to the next. On J-TTL, we find that existing adaptation methods like reflection, memory, or reinforcement learning struggle. To address the challenges posed by our benchmark, we present EvoTest, an evolutionary test-time learning framework that improves an agent without any fine-tuning or gradients-by evolving the entire agentic system after every episode. EvoTest has two roles: the Actor Agent, which plays the game, and the Evolver Agent, which analyzes the episode transcript to propose a revised configuration for the next run. This configuration rewrites the prompt, updates memory by logging effective state-action choices, tunes hyperparameters, and learns the tool-use routines. On our J-TTL benchmark, EvoTest consistently increases performance, outperforming not only reflection and memory-only baselines but also more complex online fine-tuning methods. Notably, our method is the only one capable of winning two games (Detective and Library), while all baselines fail to win any.

Figures (9)

• Figure 1: The EvoTest architecture, designed to enable test-time learning (TTL). The agent operates in a continuous Act-Evolve loop across multiple attempts at the same task. After each episode, the Evolver Agent analyzes the full trajectory transcript—rich narrative feedback to perform gradient-free, whole-system evolution on the agent's entire configuration. This allows the agentic system to self-improve on the fly, directly from its own experience at test time.
• Figure 2: Learning curves showing final score per episode across six Jericho games with google/gemini-2.5-flash as LLM. EvoTest consistently demonstrates a steeper and more stable learning trajectory compared to baselines, validating its effectiveness for test-time learning.
• Figure 2: Practical costs for a single learning update.
• Figure 3: Ablation study on EvoTest components, showing Area Under the Curve (AUC) scores.
• Figure 4: Case Study: Prompt of Episode 0.