CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

TL;DR

CLEANER tackles execution noise in agentic reinforcement learning with parameter-constrained LLMs by introducing Similarity-Aware Adaptive Rollback (SAAR), a data-level trajectory purification method that retrospectively replaces failures with self-corrections. SAAR adaptively governs shallow versus deep replacements based on semantic similarity, producing self-purified trajectories that bias learning toward correct reasoning rather than error-recovery loops. Empirically, CLEANER yields average gains of approximately $6\%$, $3\%$, and $5\%$ on AIME, GPQA, and LiveCodeBench, respectively, and matches state-of-the-art performance using only about one-third of the training steps. This approach offers a scalable, efficient alternative to dense rewards or supersampling for robust agentic RL with tool use.

Abstract

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub

Figures (5)

• Figure 1: Left: Illustration of the differences between the standard baseline and our CLEANER. Right: By reducing the number of tool execution failures within trajectories during training, our method improves pass@1 accuracy on AIME’25 by 8.1%.
• Figure 2: Impact of execution noise. Spikes in the average number of tool execution failures per trajectory correlate directly with accuracy degradation on AIME25, highlighting the sensitivity of policy optimization to error-contaminated trajectories.
• Figure 3: Illustration of our Similarity-Aware Adaptive Rollback (SAAR).
• Figure 4: Evolution of training metrics during RL. Compared to the DAPO-baseline, CLEANER effectively suppresses erroneous tool calls in trajectories, leading to significant performance gains.
• Figure 5: Recovery from suboptimal policies. Comparison of training metrics before and after introducing CLEANER at step 200. The inclusion of CLEANER effectively stabilizes the optimization process, leading to a marked improvement in final performance.