Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Process-based Self-Rewarding Language Models

Shimao Zhang, Xiao Liu, Xin Zhang, Junxiao Liu, Zheheng Luo, Shujian Huang, Yeyun Gong

TL;DR

The paper tackles the challenge of improving mathematical reasoning in LLMs beyond human preferences by introducing Process-based Self-Rewarding Language Models, which integrate long-chain step-by-step reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization into a self-rewarding loop. By initializing with EFT/IFT data, using a PRM-driven data generation pipeline, and training with step-wise Direct Preference Optimization, the approach yields progressive improvements on a suite of math benchmarks for both 7B and 72B models, and enables effective test-time scaling. The results demonstrate that fine-grained, step-level rewards and judgments can surpass traditional self-rewarding in math tasks, with evidence of stable gains across iterations and benchmarks. This work highlights the potential for self-supervised, step-wise self-improvement to push LLMs toward higher-order reasoning capabilities, potentially approaching or surpassing human performance on complex mathematical problems.

Abstract

Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.

Process-based Self-Rewarding Language Models

TL;DR

The paper tackles the challenge of improving mathematical reasoning in LLMs beyond human preferences by introducing Process-based Self-Rewarding Language Models, which integrate long-chain step-by-step reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization into a self-rewarding loop. By initializing with EFT/IFT data, using a PRM-driven data generation pipeline, and training with step-wise Direct Preference Optimization, the approach yields progressive improvements on a suite of math benchmarks for both 7B and 72B models, and enables effective test-time scaling. The results demonstrate that fine-grained, step-level rewards and judgments can surpass traditional self-rewarding in math tasks, with evidence of stable gains across iterations and benchmarks. This work highlights the potential for self-supervised, step-wise self-improvement to push LLMs toward higher-order reasoning capabilities, potentially approaching or surpassing human performance on complex mathematical problems.

Abstract

Large Language Models have demonstrated outstanding performance across various downstream tasks and have been widely applied in multiple scenarios. Human-annotated preference data is used for training to further improve LLMs' performance, which is constrained by the upper limit of human performance. Therefore, Self-Rewarding method has been proposed, where LLMs generate training data by rewarding their own outputs. However, the existing self-rewarding paradigm is not effective in mathematical reasoning scenarios and may even lead to a decline in performance. In this work, we propose the Process-based Self-Rewarding pipeline for language models, which introduces long-thought reasoning, step-wise LLM-as-a-Judge, and step-wise preference optimization within the self-rewarding paradigm. Our new paradigm successfully enhances the performance of LLMs on multiple mathematical reasoning benchmarks through iterative Process-based Self-Rewarding, demonstrating the immense potential of self-rewarding to achieve LLM reasoning that may surpass human capabilities.

Paper Structure

This paper contains 29 sections, 8 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Reinforcement Learning from Human Feedback
  4. LLM-as-a-Judge
  5. Self-Rewarding Language Models
  6. Step-by-step Reasoning
  7. Process-based Self-Rewarding Language Models
  8. Preliminary Study
  9. Model Initialization
  10. Step-by-step Long-chain Reasoning and Preference Data Generation
  11. Step-wise Model Preference Optimization
  12. Iteration Pipeline
  13. Experimental Setup
  14. Models
  15. Datasets
  16. ...and 14 more sections

Figures (6)

  • Figure 1: Illustration of our Process-based Self-Rewarding paradigm. (1) We get EFT data by tree-search, initial data filtering and data annotation. And we get IFT data by step segmentation. (2) The model is initialized on EFT and IFT data. (3) The model conducts step-by-step search-based reasoning and performs step-wise LLM-as-a-Judge to select the chosen step and generate the step-wise preference pair at each step. (4) We perform step-wise preference optimization on the model. (5) The model enters the next iteration cycle.
  • Figure 2: Prompt Distributions
  • Figure 3: Response Distributions
  • Figure 5: The prompt for converting the the given solution into step-by-step format logically without altering any information in the original solution.
  • Figure 6: The prompt for LLMs conducting step-by-step long-thought reasoning.
  • ...and 1 more figures