Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement

TL;DR

This work tackles the gap in RL-based post-tuning of large reasoning models by strengthening internal reasoning quality through self-rewriting. It introduces a selective rewriting framework that lets a model rewrite its own reasoning on simple, perfectly correct samples and learn from these rewrites within the GRPO RL loop, achieving roughly a 10% overhead. Across diverse tasks and model sizes, self-rewriting improves the accuracy-length tradeoff and markedly enhances internal reasoning quality as judged by LLMs, with gains of about $+0.6$ accuracy and $-46\%$ reasoning length, plus a $+7.2$ boost in judge scores. The approach yields more concise, diverse, and coherent reasoning while maintaining scalability, outperforming strong baselines in online RL settings and offering a flexible path for targeted reasoning improvements in future work.

Abstract

Through reinforcement learning (RL) with outcome correctness rewards, large reasoning models (LRMs) with scaled inference computation have demonstrated substantial success on complex reasoning tasks. However, the one-sided reward, focused solely on final correctness, limits its ability to provide detailed supervision over internal reasoning process. This deficiency leads to suboptimal internal reasoning quality, manifesting as issues like over-thinking, under-thinking, redundant-thinking, and disordered-thinking. Inspired by the recent progress in LRM self-rewarding, we introduce self-rewriting framework, where a model rewrites its own reasoning texts, and subsequently learns from the rewritten reasoning to improve the internal thought process quality. For algorithm design, we propose a selective rewriting approach wherein only "simple" samples, defined by the model's consistent correctness, are rewritten, thereby preserving all original reward signals of GRPO. For practical implementation, we compile rewriting and vanilla generation within one single batch, maintaining the scalability of the RL algorithm and introducing only ~10% overhead. Extensive experiments on diverse tasks with different model sizes validate the effectiveness of self-rewriting. In terms of the accuracy-length tradeoff, the self-rewriting approach achieves improved accuracy (+0.6) with substantially shorter reasoning (-46%) even without explicit instructions in rewriting prompts to reduce reasoning length, outperforming existing strong baselines. In terms of internal reasoning quality, self-rewriting achieves significantly higher scores (+7.2) under the LLM-as-a-judge metric, successfully mitigating internal reasoning flaws.

Figures (5)

• Figure 1: The reasoning of LRMs suffers from internal flaws. Over-thinking manifests as reasoning heavily over trivial or irrelevant aspects, under-thinking manifests as omissions and insufficient depth in reasoning, redundant-thinking manifests as repetition of essentially the same thoughts without bringing new ideas, and disordered-thinking manifests as jumping abruptly from one topic to another without coherence. Evaluated by LLM judges scores (the higher the better), rewriting significantly mitigates the flaws. Self-rewriting framework is to enable LRMs to learn from their own rewrites.
• Figure 2: Self-rewriting framework. If any response for a given query is incorrect, the remaining half of the rollouts are sampled normally, and the final loss computation proceeds identically to GRPO. Conversely, if all initial responses for a query are correct, the model then rewrites and completes the reasoning texts, with the rewritten version receiving higher rewards.
• Figure 3: The illustration of computation process for vanilla GRPO (top) and self-rewriting (bottom), when there are $4$ computation units, $2$ queries in a batch, and $4$ rollouts for each query. By compiling generation and rewriting into one batch, the complexity introduced by selective rewriting only leads to time consumption increment of about 10%.
• Figure 4: Results of four checkpoints for each methods on Qwen3-8B. X-axis refers to length and y-axis refers to average accuracy across four tasks.
• Figure 5: The length ratio distribution of Qwen3-8B online RL training dataset between preferred and other responses. Preferred responses refer to rewritten ones for self-rewriting, and shortest correct ones for vanilla length control approaches. The vertical dashed lines denote median values. The probability density function shown in the figure is obtained using Gaussian kernel density estimation.