SAPO: Self-Adaptive Process Optimization Makes Small Reasoners Stronger

TL;DR

This work tackles the inefficiency and misalignment of existing self-evolution methods for small language models by introducing SAPO, a self-adaptive process optimization framework that localizes the first error in reasoning and uses rollout-based verifier estimates only where needed. By replacing costly Monte Carlo rollouts with first-error detection and a principled ORPO-based self-alignment, SAPO delivers accurate, step-level supervision that narrows the reasoner–verifier gap and improves multi-step reasoning tasks in mathematics and coding for small models. The authors also contribute two process-verified benchmarks, GSM_Process and MBPP_Process, and demonstrate that SAPO outperforms prior self-evolution approaches across multiple backbones, while achieving substantial efficiency gains. The approach offers practical impact for deploying reliable, high-performing reasoning on mobile-friendly models, paving the way for more scalable and robust self-improvement pipelines in resource-constrained settings.

Abstract

Existing self-evolution methods overlook the influence of fine-grained reasoning steps, which leads to the reasoner-verifier gap. The computational inefficiency of Monte Carlo (MC) process supervision further exacerbates the difficulty in mitigating the gap. Motivated by the Error-Related Negativity (ERN), which the reasoner can localize error following incorrect decisions, guiding rapid adjustments, we propose a Self-Adaptive Process Optimization (SAPO) method for self-improvement in Small Language Models (SLMs). SAPO adaptively and efficiently introduces process supervision signals by actively minimizing the reasoner-verifier gap rather than relying on inefficient MC estimations. Extensive experiments demonstrate that the proposed method outperforms most existing self-evolution methods on two challenging task types: mathematics and code. Additionally, to further investigate SAPO's impact on verifier performance, this work introduces two new benchmarks for process reward models in both mathematical and coding tasks.

Figures (11)

• Figure 1: The performance and gap dynamics of the online reasoner and verifier across iterations in the step correction prediction of GSM_Process. The selected baseline model is Qwen-2.5-0.5B, and the reasoner estimates step correctness using a MC approach.
• Figure 2: A comparison between the Self-Adaptive Process Optimization (SAPO) and the previous self-evolution framework. SAPO actively detects potential first error positions to determine which steps need to be verified, rather than performing step-by-step rollout estimation.
• Figure 3: Overall framework diagram of Self-Adaptive Process Optimization (SAPO). The method adopts a self-iterative framework where the verifier pre-assigns step-level scores, error detection locates the first likely error, and the reasoner revisits it for posterior estimation. The corrected reasoning step labels then supervise the verifier, enabling the reasoner to self-optimize under more accurate process rewards.
• Figure 4: The performance evolution of SAPRM across different tasks (GSM_Process and MBPP_Process) and models over iterations. The online Outcome Reward Model (ORM) is used as a baseline for comparison.
• Figure 5: Multi-round iterative performance comparison of different reward models on GSM_Process. Monte Carlo Estimation (MCE) serves as the upper bound of verification performance achievable by the reasoner, and Qwen-2.5-0.5B is used as the base model.