Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop

Xuancheng Li, Haitao Li, Yujia Zhou, YiqunLiu, Qingyao Ai

TL;DR

This work tackles the sparsity of outcome-based rewards in reinforcement learning with verifiable rewards (RLVR) by injecting rich, verbal feedback into a multi-turn learning loop. MulFeRL triggers feedback-driven regeneration only on all-failed samples, and utilizes two complementary learning signals—within-turn GRPO for mixed groups and cross-turn DPO for all-positive revisions—along with structured feedback injection to embed guidance into the model's reasoning. Empirical results on OpenR1-Math and related benchmarks show substantial improvements over supervised finetuning and prior RLVR baselines, with ablations confirming the necessity of dynamic regeneration, the two learning signals, and the feedback injection mechanism. The approach also demonstrates generalization to out-of-domain reasoning tasks and reveals important insights into the impact of feedback quality and test-time multi-turn feedback, suggesting practical benefits for education and reliable reasoning systems while highlighting costs and robustness considerations that warrant further research.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in multiple domains, yet outcome-only scalar rewards are often sparse and uninformative, especially on failed samples, where they merely indicate failure and provide no insight into why the reasoning fails. In this paper, we investigate how to leverage richer verbal feedback to guide RLVR training on failed samples, and how to convert such feedback into a trainable learning signal. Specifically, we propose a multi-turn feedback-guided reinforcement learning framework. It builds on three mechanisms: (1) dynamic multi-turn regeneration guided by feedback, triggered only on failed samples, (2) two complementary learning signals for within-turn and cross-turn optimization, and (3) structured feedback injection into the model's reasoning process. Trained on sampled OpenR1-Math, the approach outperforms supervised fine-tuning and RLVR baselines in-domain and generalizes well out-of-domain.

MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loop

TL;DR

This work tackles the sparsity of outcome-based rewards in reinforcement learning with verifiable rewards (RLVR) by injecting rich, verbal feedback into a multi-turn learning loop. MulFeRL triggers feedback-driven regeneration only on all-failed samples, and utilizes two complementary learning signals—within-turn GRPO for mixed groups and cross-turn DPO for all-positive revisions—along with structured feedback injection to embed guidance into the model's reasoning. Empirical results on OpenR1-Math and related benchmarks show substantial improvements over supervised finetuning and prior RLVR baselines, with ablations confirming the necessity of dynamic regeneration, the two learning signals, and the feedback injection mechanism. The approach also demonstrates generalization to out-of-domain reasoning tasks and reveals important insights into the impact of feedback quality and test-time multi-turn feedback, suggesting practical benefits for education and reliable reasoning systems while highlighting costs and robustness considerations that warrant further research.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in multiple domains, yet outcome-only scalar rewards are often sparse and uninformative, especially on failed samples, where they merely indicate failure and provide no insight into why the reasoning fails. In this paper, we investigate how to leverage richer verbal feedback to guide RLVR training on failed samples, and how to convert such feedback into a trainable learning signal. Specifically, we propose a multi-turn feedback-guided reinforcement learning framework. It builds on three mechanisms: (1) dynamic multi-turn regeneration guided by feedback, triggered only on failed samples, (2) two complementary learning signals for within-turn and cross-turn optimization, and (3) structured feedback injection into the model's reasoning process. Trained on sampled OpenR1-Math, the approach outperforms supervised fine-tuning and RLVR baselines in-domain and generalizes well out-of-domain.
Paper Structure (76 sections, 12 equations, 13 figures, 9 tables)

This paper contains 76 sections, 12 equations, 13 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Enhancing LLM Reasoning with Reinforcement Learning.
  4. Learning from Verbal Feedback.
  5. Method
  6. Problem Formulation
  7. Overview of MulFeRL
  8. Dynamic Multi-turn Feedback-driven Regeneration
  9. Two Complementary Learning Signals
  10. Structured Feedback Injection
  11. Experiment
  12. Experiment Settings
  13. Datasets.
  14. Baselines.
  15. Implementation Details.
  16. ...and 61 more sections

Figures (13)

  • Figure 1: Illustration of MulFeRL.
  • Figure 2: The output format used by MulFeRL.
  • Figure 3: Test-time multi-turn feedback results on TheoremQA (Qwen2.5-7B-Base). We vary the number of feedback turns.
  • Figure 4: Training efficiency (Qwen3-4B-Inst). (a) Training cost across methods. (b) MulFeRL efficiency across training steps: average training time per step and the average number of regenerated samples per step in Turn-1/Turn-2, reported as averages over 50-step intervals.
  • Figure 5: Impact of training turns on both effectiveness and efficiency (Qwen3-4B-Inst). We vary the number of training turns and report the best performance and the average training time.
  • ...and 8 more figures