Expanding the Capabilities of Reinforcement Learning via Text Feedback

TL;DR

This work addresses the inefficiency of sparse scalar rewards in reinforcement learning for large language models by introducing RL from Text Feedback (RLTF), where text critiques guide learning during training but are unavailable at test time. It proposes two complementary methods: Self Distillation (RLTF-SD), which distills feedback-conditioned second-turn solutions into a stronger single-turn policy, and Feedback Modeling (RLTF-FM), which learns from the critiques themselves as an auxiliary objective and enables test-time self-critique. The authors provide theoretical analyses showing reward-only signals can be rare and poorly conditioned, whereas feedback signals offer a better-conditioned representation for learning, and they demonstrate empirical gains across reasoning, mathematics, and creative writing benchmarks. Test-time scaling via self-feedback further illustrates how RLTF-FM can yield improvements without external judges. Overall, RLTF demonstrates that abundant textual feedback can scale RL effectiveness beyond scalar rewards and demonstrations, with practical implications for aligning and improving LLM behavior.

Abstract

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study text feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, annotators, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.

Figures (6)

• Figure 1: Left: overview of reinforcement learning from text feedback, which uses a feedback provider (judge) to generate critiques $c_0$. RLTF-SD trains the policy to match the feedback-conditioned second-turn generations $y_1$, and RLTF-FM predicts the critiques $c_0$ as an auxiliary objective. Right: performance of our two methods, Self Distillation (RLTF-SD) and Feedback Modeling (RLTF-FM), on reasoning puzzles, competition math, and creative writing tasks. Both methods outperform standard single-turn GRPO.
• Figure 2: Evaluation curves on Knights and Knaves and MATH500 (trained on DAPO) for ablations on self distillation design choices. For each environment: Left: single-turn accuracy; Right: multi-turn accuracy. RLTF-SD (GRPO baseline) denotes using AWR objective with second turn mean baseline. RLTF-SD (PPO clipping) denotes using PPO style clipping on importance weighting with first turn baseline. RLTF-SD (CISPO clipping) denotes using CISPO style clipping on importance weighting with first turn baseline. Note that our proposed design choices consistently outperform the alternatives in both single-turn and multi-turn performance.
• Figure 3: Evaluation curves on Knights and Knaves and MATH500 (trained on DAPO) for text feedback vs. correctness-only feedback. We compare single- and multi-turn accuracy on two algorithms: multi-turn GRPO and RLTF-SD. Overall, using text feedback outperforms using correctness-only feedback for single-turn and multi-turn accuracy on both algorithms.
• Figure 4: Test-time scaling results on Knights and Knaves and MATH500 (trained on DAPO). We allow the model to generate multiple rounds of self-feedback at inference time (denoted in the x-axis). We compare RLTF-FM with multi-turn scalar-based RL, and the dashed line ("+ Self-Critique") denotes further using RL to improve the self-critique during training (alg:feedback_modeling_self_critique). We use skipped y-axis for the plot on the left for ease of presentation.
• Figure 5: Evaluation curves for single-turn accuracy across reasoning puzzles, competition math, and creative writing tasks, at every 40 training steps. For reasoning tasks, we report the mean@1 accuracy judged by either verifiable reward or LLM-as-a-judge. For math tasks, we report the mean@1 accuracy. For Litbench, we report the mean@1 accuracy judged by LLM-as-a-judge. The accuracy in reasoning and math is normalized between 0 and 1, and the score in creative writing is normalized between 1 and 10.

Theorems & Definitions (30)

• remark 1
• proposition 1: Reward-only bottlenecks under base rollouts (informal)
• proposition 2: Feedback modeling yields a well-conditioned representation signal (informal)
• proposition 3: In-sample second-turn group-mean baseline yields $(1-\tfrac{1}{N})$ shrinkage
• proof
• proposition 4: First-turn group-mean baseline is unbiased (with IS correction)
• proof
• proposition 5: Deterministic collapse under second-turn mean baselines
• proof
• proposition 6: How often collapse occurs for Bernoulli rewards