Reinforcement Learning via Self-Distillation

TL;DR

This work formalizes reinforcement learning with rich feedback (RLRF) and introduces Self-Distillation Policy Optimization (SDPO), where the current policy acts as a self-teacher conditioned on tokenized feedback to provide dense, per-token credit signals without external supervision. SDPO preserves on-policy exploration while distilling retrospective guidance, enabling faster learning and more concise reasoning across reasoning, tool use, and competitive-programming tasks, with gains growing as model strength increases. Beyond standard RLVR improvements, SDPO also boosts test-time discovery on hard binary-reward problems by compressing experience into model weights, achieving substantial speedups. The approach is scalable, largely drop-in for existing RLVR pipelines, and highlights a principled way to leverage in-context retrospection as a form of self-supervision for credit assignment in LLMs.

Abstract

Large language models are increasingly post-trained with reinforcement learning in verifiable domains such as code and math. Yet, current methods for reinforcement learning with verifiable rewards (RLVR) learn only from a scalar outcome reward per attempt, creating a severe credit-assignment bottleneck. Many verifiable environments actually provide rich textual feedback, such as runtime errors or judge evaluations, that explain why an attempt failed. We formalize this setting as reinforcement learning with rich feedback and introduce Self-Distillation Policy Optimization (SDPO), which converts tokenized feedback into a dense learning signal without any external teacher or explicit reward model. SDPO treats the current model conditioned on feedback as a self-teacher and distills its feedback-informed next-token predictions back into the policy. In this way, SDPO leverages the model's ability to retrospectively identify its own mistakes in-context. Across scientific reasoning, tool use, and competitive programming on LiveCodeBench v6, SDPO improves sample efficiency and final accuracy over strong RLVR baselines. Notably, SDPO also outperforms baselines in standard RLVR environments that only return scalar feedback by using successful rollouts as implicit feedback for failed attempts. Finally, applying SDPO to individual questions at test time accelerates discovery on difficult binary-reward tasks, achieving the same discovery probability as best-of-k sampling or multi-turn conversations with 3x fewer attempts.

Figures (23)

• Figure 1: SDPO substantially outperforms an improved version of Group Relative Policy Optimization (GRPO) on LCB v6 with Qwen3-8B. Further, SDPO achieves GRPO's final accuracy in $4\times$ fewer generations. Claude Sonnet 4 is the strongest instruct model on the public LCBv6 leaderboard. Shaded regions show the standard deviation across 3 seeds.
• Figure 2: Comparison of RLVR and RLRF settings. In Reinforcement Learning with Verifiable Rewards (RLVR), the agent learns from a scalar reward $r$, which often acts as an information bottleneck by masking the underlying environment state. In contrast, Reinforcement Learning with Rich Feedback (RLRF) utilizes tokenized feedback. This provides a significantly richer signal than a scalar reward, as the feedback can encapsulate both the reward as well as detailed observations of the state (such as runtime errors from a code environment or feedback from an LLM judge).
• Figure 3: Example of feedback from our code environment, inspired by LeetCode. Listings \ref{['lst:feedback_example_wrong_answer']}, \ref{['lst:memory_error']}, and \ref{['lst:index_error']} in the appendix show examples of feedback in case of a wrong answer, a memory error, and an index error.
• Figure 4: Example of self-teaching with Qwen3-8B. The answer is generated by the model before seeing the feedback. Then, we re-evaluate the log-probs of the original attempt with the self-teacher after seeing the feedback. We show the per-token $\log(\mathbb{P}(*){\text{self-teacher}}/\mathbb{P}(*){\text{student}})$, with red indicating negative values (self-teacher disagrees) and white indicating values around zero. Notably, in this example, Qwen3-8B identifies the error through retrospection without an explicit solution. Further, the activation is sparse, identifying where mistakes happen and adjusting to the students' response distribution.
• Figure 5: Time per step for SDPO vs GRPO (solid: without code environment, light: with code environment).

Theorems & Definitions (7)

• Proposition 2.1
• Definition 5.1: Discovery time
• proof : Proof of Proposition \ref{['prop:gradient_estimate']}
• Proposition B.1
• proof
• Proposition B.2: EMA yields an approximate trust region
• proof