R$^3$L: Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification

TL;DR

R^3L tackles core RL bottlenecks in language-guided, multi-turn tasks by integrating language-guided Reflect-then-Retry for active trajectory synthesis, Pivotal Credit Assignment to localize learning signals to diverging suffixes, and Positive Amplification to ensure constructive gradients dominate off-policy updates. The method achieves robust gains across agentic environments and mathematical reasoning benchmarks, outperforming baselines like GRPO, GSPO, and Critique-GRPO while maintaining training stability. Through extensive ablations and analyses, the paper demonstrates that active trajectory synthesis, precise credit assignment, and gradient shaping are crucial for effective learning in sparse-reward, long-horizon tasks. The work provides a practical, scalable framework and releases code to facilitate replication and further research in RL for LLM-driven agents.

Abstract

Reinforcement learning drives recent advances in LLM reasoning and agentic capabilities, yet current approaches struggle with both exploration and exploitation. Exploration suffers from low success rates on difficult tasks and high costs of repeated rollouts from scratch. Exploitation suffers from coarse credit assignment and training instability: Trajectory-level rewards penalize valid prefixes for later errors, and failure-dominated groups overwhelm the few positive signals, leaving optimization without constructive direction. To this end, we propose R$^3$L, Reflect-then-Retry Reinforcement Learning with Language-Guided Exploration, Pivotal Credit, and Positive Amplification. To synthesize high-quality trajectories, R$^3$L shifts from stochastic sampling to active synthesis via reflect-then-retry, leveraging language feedback to diagnose errors, transform failed attempts into successful ones, and reduce rollout costs by restarting from identified failure points. With errors diagnosed and localized, Pivotal Credit Assignment updates only the diverging suffix where contrastive signals exist, excluding the shared prefix from gradient update. Since failures dominate on difficult tasks and reflect-then-retry produces off-policy data, risking training instability, Positive Amplification upweights successful trajectories to ensure positive signals guide the optimization process. Experiments on agentic and reasoning tasks demonstrate 5\% to 52\% relative improvements over baselines while maintaining training stability. Our code is released at https://github.com/shiweijiezero/R3L.

Figures (14)

• Figure 1: Comparison between standard RL (GRPO) and R$^3$L. Red blocks indicate erroneous steps, Green blocks indicate correct steps, and Gray blocks indicate masked prefix excluded from gradient updates. Standard RL suffers from (C1) inefficient stochastic sampling, (C2) valid prefix penalization, and (C3) gradient asymmetry due to failure dominance. R$^3$L addresses these via (S1) reflect-then-retry for active exploration, (S2) pivotal credit, and (S3) positive amplification. The detailed R$^3$L framework is illustrated in Figure \ref{['fig:architecture']}.
• Figure 2: Overview of the R$^3$L framework. The method utilizes Language-Guided Reflect-Then-Retry to synthesize high-reward trajectories via verbal feedback. To preserve valid steps, Pivotal Credit Assignment masks shared prefixes to isolate critical decision points, while Positive Amplification scales the advantages of successful trajectories to stabilize off-policy training.
• Figure 3: Evolution of exploration metrics across environments by the Reflect-then-Retry mechanism. (a) The average Reward Gain of retry trajectory relative to the base trajectory. (b) The percentage of retry trajectories that successfully improved upon the base attempt.
• Figure 4: Training Dynamics and Stability Analysis on ALFWorld. (a) Average reward comparison. (b) Reference KL divergence between current policy and reference policy. (c) Gradient norm during training. (d) Policy loss comparison. (e) GRPO update KL between new and old policy. (f) GRPO clip fraction.
• Figure 5: Four types of trajectories in R$^3$L. Type 1 represents base exploration from the current policy. Type 2 captures the reflection process that diagnoses errors and identifies pivot points. Type 3 shows retry generation conditioned on diagnostic guidance. Type 4 is the distillation trajectory that combines the original prefix with the corrected suffix, removing guidance dependency for training.

Theorems & Definitions (13)

• Definition 1: Gradient Decomposition
• Proposition 2: Entropy Collapse
• proof
• Theorem 3: Gradient Dominance Condition
• proof
• Corollary 4: Robustness of $\alpha = 3.0$
• Theorem 5: Variance Reduction
• proof
• Proposition 6: Covariance Analysis
• Theorem 7: Off-Policy Stability