A Stitch in Time Saves Nine: Proactive Self-Refinement for Language Models

TL;DR

This paper addresses the limitations of reactive self-refinement in large language models by introducing ProActive Self-Refinement (PASR), which refines outputs during generation. PASR employs a reinforcement learning framework using Group Relative Policy Optimization and a comparison-based reward to decide when and how to refine, with a structured generation trace conveyed through <think>, <refine>, and <answer> tags. Evaluations across ten open-domain tasks on Qwen backbones show substantial gains in efficiency and accuracy, notably a 41.6% reduction in token usage and an 8.2% accuracy improvement on Qwen3-8B. The approach demonstrates robust generalization beyond domain-specific data and provides open-source code and data for replication and further development.

Abstract

Recent advances in self-refinement have demonstrated significant potential for improving the outputs of large language models (LLMs) through iterative refinement. However, most existing self-refinement methods rely on a reactive process with a fixed number of iterations, making it difficult to determine the optimal timing and content of refinement based on the evolving generation context. Inspired by the way humans dynamically refine their thoughts during execution, we propose ProActive Self-Refinement (PASR), a novel method that enables LLMs to refine their outputs during the generation process. Unlike methods that regenerate entire responses, PASR proactively decides whether, when, and how to refine based on the model's internal state and evolving context. We conduct extensive experiments on a diverse set of 10 tasks to evaluate the effectiveness of PASR. Experimental results show that PASR significantly enhances problem-solving performance. In particular, on Qwen3-8B, PASR reduces average token consumption by 41.6% compared to standard generation, while also achieving an 8.2% improvement in accuracy. Our code and baselines used in the paper are available on GitHub.

Figures (5)

• Figure 1: Comparison between the post-hoc refinement method (middle) and our proposed PASR (right). The post-hoc refinement method iteratively refines its initial answer. In contrast, PASR proactively refines its reasoning process during the generation.
• Figure 2: Left: Answer format used in PASR. Right: Reward design for a generated answer $y^{'}$during training. The total reward is computed as the sum of the format score, accuracy score, and refinement score, as defined in Equation \ref{['overall_reward']}.
• Figure 3: Comparison of average token length across different methods on various tasks. The left figure uses the Qwen3-8B backbone, while the right figure uses Qwen2.5-7B.
• Figure 4: From left to right, the pie charts show: (1) the proportion of answers changed by PASR refinement, (2) the distribution of coherence scores reflecting how well the self-refinement builds upon the initial generation, and, and (3) the distribution of alignment scores measuring the consistency between the refinement process and the final answer. For (2) and (3), each segment represents the proportion of examples falling within a specific score range (e.g., [0–0.45), [0.45–0.85), [0.85–1.0]).
• Figure 5: The frequency distribution of the four refinement types in PASR.