Self-Improving LLM Agents at Test-Time

TL;DR

This work introduces TT-SI, a test-time, uncertainty-guided framework enabling agentic LLMs to learn from challenging instances on the fly. It combines a self-awareness module (uncertainty estimation), a self-augmentation module (on-the-fly data generation), and a self-learning module (lightweight test-time fine-tuning, e.g., LoRA) to adapt per instance, yielding significant accuracy gains with far fewer data. The approach outperforms standard inductive learning baselines across multiple tool-use benchmarks and even offers training-free ICL alternatives, suggesting a practical path toward self-evolving agents. The study also discusses limitations and future directions, including threshold selection, data-generator co-evolution, and domain-specific extensions.

Abstract

One paradigm of language model (LM) fine-tuning relies on creating large training datasets, under the assumption that high quantity and diversity will enable models to generalize to novel tasks after post-training. In practice, gathering large sets of data is inefficient, and training on them is prohibitively expensive; worse, there is no guarantee that the resulting model will handle complex scenarios or generalize better. Moreover, existing techniques rarely assess whether a training sample provides novel information or is redundant with the knowledge already acquired by the model, resulting in unnecessary costs. In this work, we explore a new test-time self-improvement method to create more effective and generalizable agentic LMs on-the-fly. The proposed algorithm can be summarized in three steps: (i) first it identifies the samples that model struggles with (self-awareness), (ii) then generates similar examples from detected uncertain samples (self-data augmentation), and (iii) uses these newly generated samples at test-time fine-tuning (self-improvement). We study two variants of this approach: Test-Time Self-Improvement (TT-SI), where the same model generates additional training examples from its own uncertain cases and then learns from them, and contrast this approach with Test-Time Distillation (TT-D), where a stronger model generates similar examples for uncertain cases, enabling student to adapt using distilled supervision. Empirical evaluations across different agent benchmarks demonstrate that TT-SI improves the performance with +5.48% absolute accuracy gain on average across all benchmarks and surpasses other standard learning methods, yet using 68x less training samples. Our findings highlight the promise of TT-SI, demonstrating the potential of self-improvement algorithms at test-time as a new paradigm for building more capable agents toward self-evolution.

Figures (13)

• Figure 1: Overview of the Test-Time Self-Improvement (TT-SI) framework.(Left) TT-SI enables on-the-fly adaptation by targeting uncertain test instances during inference. It consists of three steps: (1) Self-Awareness: An Uncertainty Estimator ($\mathcal{\textbf{H}}$) identifies challenging samples. (2) Self-Data Augmentation: For each identified uncertain sample, one similar variant is automatically generated using Data Synthesis Function ($\mathcal{\textbf{G}}$). (3) Self-Improvement:Test-Time Fine-tuning ($\mathcal{\textbf{T}}$) applies a lightweight update using only one generated training instance per case. (Right)$\Delta$-accuracy gains of TT-SI over the prompting baseline at test-time. TT-SI improves the baseline by +5.48% on average across ToolAlpaca (+5.84%), NexusRaven (+6.05%), SealTool (+5.76%), and API-Bank (+4.26%).
• Figure 2: Experimental Results on SealTool.Left: Accuracy comparison of TT-SI against standard baselines (left-most) and its variants (middle), including ablations (right-most). Right: Scaling behavior under different adaptation strategies with varying numbers of samples. Shaded regions show variance over five runs; dashed lines denote baseline references.
• Figure 3: Model Scaling. Model Scale and Architectural Generalization with Qwen2.5-7B-Instruct.
• Figure 4: Impact of Uncertainty. Performance of TT-SI when trained on certain samples (w. $\mathcal{\textbf{H}}$'), uncertain samples (main), and all samples (w/o $\mathcal{\textbf{H}}$).
• Figure 5: Comparison of PPL and RSS (ours) as Uncertainty Estimator ($\mathcal{\textbf{H}}$) on SealTool. Histograms show score differences between the top-1 and top-2 predictions. The green bars denote correct predictions, and red bars denote incorrect ones. PPL-based uncertainty (left) shows strong overlap between correct and incorrect cases, whereas our RSS-based estimator (right) yields a clearer separation, enabling more reliable uncertainty filtering.