Learning to Reason via Self-Iterative Process Feedback for Small Language Models

TL;DR

This paper tackles the reasoning limitations of open-source small language models by introducing self-iterative process feedback (SIPF). SIPF leverages a lightweight simulator and verifier to assign granular, step-level correctness signals and uses odds-ratio preference optimization (ORPO) to align models on diverse, self-generated reasoning paths, guided by a process reward model (PRM). Empirical results on GSM8K, MBPP, MMLU_Math, and HumanEval show substantial in-domain gains and notable out-of-domain generalization, with Gemma-2B achieving a 12.43 point ACC increase on GSM8K and a 3.95 point Pass@1 on MBPP after one iteration. The work demonstrates that process-level feedback, rather than only outcome accuracy, yields more robust reasoning improvements for open-source LMs, while also providing insights into reward-model selection and ablation analyses.

Abstract

Small language models (SLMs) are more efficient, cost-effective, and customizable than large language models (LLMs), though they often underperform in specific areas like reasoning. Past methods for enhancing SLMs' reasoning, such as supervised fine-tuning and distillation, often depend on costly external signals, resulting in SLMs being overly confident with limited supervision signals, thus limiting their abilities. Therefore, this study enables SLMs to learn to reason from self-iterative feedback. By combining odds ratio preference optimization (ORPO), we fine-tune and align SLMs using positive and negative signals generated by themselves. Additionally, we introduce process supervision for rewards in preference alignment by sampling-based inference simulation and process reward models. Compared to Supervised Fine-Tuning (SFT), our method improves the performance of Gemma-2B by 12.43 (Acc) on GSM8K and 3.95 (Pass@1) on MBPP. Furthermore, the proposed method also demonstrated superior out-of-domain generalization capabilities on MMLU_Math and HumanEval.

Figures (10)

• Figure 1: The conceptual diagram of the proposed self-iterative process feedback method against several previous methods (below). Compared to prior approaches, SIPF emphasizes on the correctness of reasoning steps. This means that SIPF can distinguish between correct reasoning with incorrect results and incorrect reasoning with correct results (above).
• Figure 2: Overall framework of the proposed learning to reason from self-iterative process feedback. A single iteration of the online learning process includes sampling, collecting, inference simulation, fine-tuning verifier, scoring to construct the preference dataset, and RL alignment by ORPO.
• Figure 3: Comparison of the training processes of SFT-based self-taught, DPO-based SRF, and ORPO-based SIPF on GSM8K using Gemma-2B. The DPO-based SRF consistently reduced the probability of generating ${\tau^w}$. SIPF is more effective than self-taught methods at reducing the probability of generating ${\tau^l}$. Additionally, different types of feedback (outcome or process) have different effects on training.
• Figure 4: Comparison of accuracy and data count for various self-iteration methods across different iterations on GSM8K.
• Figure 5: Automatic evaluation of the reliability of rationales generated by different methods (process-based or outcome-based) using GPT-4.