Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following

Kongcheng Zhang, Qi Yao, Shunyu Liu, Wenjian Zhang, Min Cen, Yang Zhou, Wenkai Fang, Yiru Zhao, Baisheng Lai, Mingli Song

TL;DR

The paper tackles sparse reward problems in reinforcement learning for complex instruction following by introducing Hindsight Instruction Replay (HiR), which selects diverse failure cases and rewrites them into hindsight instructions for replay under a binary reward. The method unifies response- and instruction-level preferences within a dual-preference learning framework, enabling efficient optimization with limited signal. Empirically, HiR improves performance across multiple backbones and instruction benchmarks, including out-of-domain reasoning tasks, and shows notable gains for weaker models while preserving general reasoning capabilities. This approach enhances sample efficiency and learning stability, with potential for broader multi-modal and agentic applications.

Abstract

Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and dataset is available at https://github.com/sastpg/HIR.

Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following

TL;DR

The paper tackles sparse reward problems in reinforcement learning for complex instruction following by introducing Hindsight Instruction Replay (HiR), which selects diverse failure cases and rewrites them into hindsight instructions for replay under a binary reward. The method unifies response- and instruction-level preferences within a dual-preference learning framework, enabling efficient optimization with limited signal. Empirically, HiR improves performance across multiple backbones and instruction benchmarks, including out-of-domain reasoning tasks, and shows notable gains for weaker models while preserving general reasoning capabilities. This approach enhances sample efficiency and learning stability, with potential for broader multi-modal and agentic applications.

Abstract

Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints. Despite the encouraging results, RL improvement inevitably relies on sampling successful, high-quality responses; however, the initial model often struggles to generate responses that satisfy all constraints due to its limited capabilities, yielding sparse or indistinguishable rewards that impede learning. In this work, we propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks, which employs a select-then-rewrite strategy to replay failed attempts as successes based on the constraints that have been satisfied in hindsight. We perform RL on these replayed samples as well as the original ones, theoretically framing the objective as dual-preference learning at both the instruction- and response-level to enable efficient optimization using only a binary reward signal. Extensive experiments demonstrate that the proposed HiR yields promising results across different instruction following tasks, while requiring less computational budget. Our code and dataset is available at https://github.com/sastpg/HIR.
Paper Structure (23 sections, 1 theorem, 17 equations, 6 figures, 6 tables, 2 algorithms)

This paper contains 23 sections, 1 theorem, 17 equations, 6 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Instruction Following
  4. Evaluation Metrics
  5. Hindsight Instruction Replay
  6. Select-then-Rewrite Replay Strategy
  7. Reinforcement Learning Objective
  8. Theoretical Perspective
  9. Experiments
  10. Experimental Setup
  11. Main Results
  12. Ablation Study
  13. In-Depth Analysis
  14. Related Work
  15. Conclusion
  16. ...and 8 more sections

Key Result

Proposition 3.2

The HiR objective is a form of preference learning on both the response- and instruction-level. where $y^w$ and $y^l$ denote the winning (positive) and losing (negative) responses, $y^r$ denotes the responses that are selected for replay, $\alpha_1, \alpha_2, \beta_1, \beta_2$ are all positive values calculated based on the rewards of samples.

Figures (6)

  • Figure 1: (Left) A conceptual illustration of the sparse and indistinguishable reward problem in current RLVR methods for instruction following tasks. (Right) Performance comparison between small LLMs trained by HiR and frontier LLMs on different benchmarks.
  • Figure 2: The overall framework of HiR with a select-then-rewrite replay strategy. First, we generate samples and select valuable failure attempts for replay with a curriculum schedule. Then we rewrite the instructions of selected samples into "hindsight" pseudo-instructions by removing the unmet constraints. Finally, we perform RL on both replayed samples as well as the original ones.
  • Figure 3: (a) The pass@$k$ curves comparison after training, and (b) constraint-level accuracy heatmap comparison during training.
  • Figure 4: Ablation study of the initial curriculum weight, with red markers indicating the best performance for each model.
  • Figure 5: Training curves of different model backbones. HiR exhibits higher training efficiency than baseline RL-CR.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Definition 3.1
  • Proposition 3.2
  • proof