Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Saiyong Yang, Yunfang Wu

TL;DR

The paper addresses exploration stagnation in reinforcement learning with verifiable rewards (RLVR) for large language models by introducing LTE, a method that uses the LLM’s own previous incorrect answers as hints to guide additional, targeted rollouts without external guidance. LTE enriches learning through hinted extra rollouts for none-pass samples and employs mixed-policy optimization with off-policy importance sampling to integrate these hints into training, improving both exploitation and exploration. Empirical results across six mathematics benchmarks show that LTE significantly enhances performance (e.g., +6.38 in Pass@1 and +9.00 in Pass@k on average for Qwen3-4B-Base with entropy loss) and reduces none-pass incidences during training, while maintaining a healthy level of entropy and longer, more thoughtful responses. The work demonstrates that internal exploration signals can be elicited through self-generated failures, offering a scalable, guidance-free approach with potential applicability beyond mathematics to a broader set of tasks.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of large language models (LLMs) recently. However, existing RLVR approaches merely train LLMs based on their own generated responses and are constrained by the initial capability of LLMs, thus prone to exploration stagnation, in which LLMs fail to solve more training problems and cannot further learn from the training data. Some work tries to address this by leveraging off-policy solutions to training problems but requires external guidance from experts which suffers from limited availability. In this work, we propose LTE (Learning to reason from Trial and Error), an approach hinting LLMs with their previously self-generated incorrect answers and problem of overlong responses, which does not require any external expert guidance. Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 6.38 in Pass@1 and 9.00 in Pass@k on average across six mathematics benchmarks for Qwen3-4B-Base. Further analysis confirms that LTE successfully mitigates the problem of exploration stagnation and enhances both exploitation and exploration during training.

Do Not Step Into the Same River Twice: Learning to Reason from Trial and Error

TL;DR

The paper addresses exploration stagnation in reinforcement learning with verifiable rewards (RLVR) for large language models by introducing LTE, a method that uses the LLM’s own previous incorrect answers as hints to guide additional, targeted rollouts without external guidance. LTE enriches learning through hinted extra rollouts for none-pass samples and employs mixed-policy optimization with off-policy importance sampling to integrate these hints into training, improving both exploitation and exploration. Empirical results across six mathematics benchmarks show that LTE significantly enhances performance (e.g., +6.38 in Pass@1 and +9.00 in Pass@k on average for Qwen3-4B-Base with entropy loss) and reduces none-pass incidences during training, while maintaining a healthy level of entropy and longer, more thoughtful responses. The work demonstrates that internal exploration signals can be elicited through self-generated failures, offering a scalable, guidance-free approach with potential applicability beyond mathematics to a broader set of tasks.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has significantly boosted the reasoning capability of large language models (LLMs) recently. However, existing RLVR approaches merely train LLMs based on their own generated responses and are constrained by the initial capability of LLMs, thus prone to exploration stagnation, in which LLMs fail to solve more training problems and cannot further learn from the training data. Some work tries to address this by leveraging off-policy solutions to training problems but requires external guidance from experts which suffers from limited availability. In this work, we propose LTE (Learning to reason from Trial and Error), an approach hinting LLMs with their previously self-generated incorrect answers and problem of overlong responses, which does not require any external expert guidance. Experiments validate the effectiveness of LTE, which outperforms the normal group relative policy optimization (GRPO) by 6.38 in Pass@1 and 9.00 in Pass@k on average across six mathematics benchmarks for Qwen3-4B-Base. Further analysis confirms that LTE successfully mitigates the problem of exploration stagnation and enhances both exploitation and exploration during training.

Paper Structure

This paper contains 28 sections, 25 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Reinforcement Learning with Verifiable Rewards (RLVR)
  4. Exploration Stagnation
  5. Method: LTE
  6. Hinted Extra Rollouts
  7. Mixed-policy Optimization
  8. Experimental Setup
  9. Language Models
  10. Training
  11. Evaluation
  12. Baselines and Methods
  13. Results and Analysis
  14. Main Results
  15. Performance on the Training Data
  16. ...and 13 more sections

Figures (4)

  • Figure 1: The framework of LTE. For none-pass samples, we extract the LLM's generated incorrect answers as hints for the extra rollouts. We do nothing to other samples which are omitted in the figure for simplicity.
  • Figure 2: The normal and hinted prompt templates used in this work.
  • Figure 3: Performance of Qwen3-4B-Base on the training data, smoothed with exponential moving average.
  • Figure 4: Training dynamics of Qwen3-4B-Base, with entropy and length smoothed by exponential moving average.

Theorems & Definitions (3)

  • Definition A.1: Reasoning State Space
  • Definition A.2: Minimums in Reasoning Space
  • Definition A.3: Failure and Pruned Subspace