S^3cMath: Spontaneous Step-level Self-correction Makes Large Language Models Better Mathematical Reasoners

TL;DR

This work introduces S$^3$c-Math, an approach to endow large language models with spontaneous step-level self-correction for mathematical reasoning. It constructs a large self-correction data corpus, S$^3$c-MathQA (532K samples), by sampling erroneous steps from existing step-by-step data and enriching them with reflection and improvement annotations, then trains models with a loss-masked instruction-tuning regime that ignores erroneous steps. Across multiple foundation LLMs, the method yields consistent improvements on GSM8K, MATH, SVAMP, and Mathematics benchmarks, validating the viability of intrinsic real-time self-correction for math problem solving. The results are supported by extensive ablations showing the superiority of step-level correction, the value of reflecting/improving content, and the effectiveness of sampling from existing paths over Monte Carlo tree search-based generation, indicating practical gains for robust mathematical reasoning in LLMs.

Abstract

Self-correction is a novel method that can stimulate the potential reasoning abilities of large language models (LLMs). It involves detecting and correcting errors during the inference process when LLMs solve reasoning problems. However, recent works do not regard self-correction as a spontaneous and intrinsic capability of LLMs. Instead, such correction is achieved through post-hoc generation, external knowledge introduction, multi-model collaboration, and similar techniques. In this paper, we propose a series of mathematical LLMs called S$^3$c-Math, which are able to perform Spontaneous Step-level Self-correction for Mathematical reasoning. This capability helps LLMs to recognize whether their ongoing inference tends to contain errors and simultaneously correct these errors to produce a more reliable response. We proposed a method, which employs a step-level sampling approach to construct step-wise self-correction data for achieving such ability. Additionally, we implement a training strategy that uses above constructed data to equip LLMs with spontaneous step-level self-correction capacities. Our data and methods have been demonstrated to be effective across various foundation LLMs, consistently showing significant progress in evaluations on GSM8K, MATH, and other mathematical benchmarks. To the best of our knowledge, we are the first to introduce the spontaneous step-level self-correction ability of LLMs in mathematical reasoning.

Figures (4)

• Figure 1: Different ways to implement self-correction. (a) Using more capable LLMs like GPT4 or a trained model to generate the feedback; (b) Apply such process at the step-level; (c) Our work, the LLMs spontaneously identify errors in their output and correct them immediately. Red text denotes the wrong step, and green text denotes the content after correction.
• Figure 2: An overview of our works. The left part of this figure illustrates the process of constructing S$^3$c-MathQA, and the right part depicts the training of S$^3$c-Math and the procedure during its reasoning.
• Figure 3: An example from S$^3$c-MathQA. Steps 1, 4, 5, and 6 are copied from the correct steps already exist in MetaMathQA. Step 2 is a wrong step sampled from the method we proposed. Step 3 is the indicator of LLM's error realization, reflection, improvement, and the new corrected step. During training, we do not compute the loss for the red texts.
• Figure 4: The prompts we used to generate reflection and improvements for S$^3$c-MathQA.