Enhancing Mathematical Reasoning in LLMs by Stepwise Correction

TL;DR

A novel prompting method named Stepwise Correction (StepCo) is proposed that helps LLMs identify and revise incorrect steps in their generated reasoning paths that iterates verification and revision phases that employ a process-supervised verifier.

Abstract

Best-of-N decoding methods instruct large language models (LLMs) to generate multiple solutions, score each using a scoring function, and select the highest scored as the final answer to mathematical reasoning problems. However, this repeated independent process often leads to the same mistakes, making the selected solution still incorrect. We propose a novel prompting method named Stepwise Correction (StepCo) that helps LLMs identify and revise incorrect steps in their generated reasoning paths. It iterates verification and revision phases that employ a process-supervised verifier. The verify-then-revise process not only improves answer correctness but also reduces token consumption with fewer paths needed to generate. With StepCo, a series of LLMs demonstrate exceptional performance. Notably, using GPT-4o as the backend LLM, StepCo achieves an average accuracy of 94.1 across eight datasets, significantly outperforming the state-of-the-art Best-of-N method by +2.4, while reducing token consumption by 77.8%.

Figures (18)

• Figure 1: StepCo mitigates the repetition of mistakes.
• Figure 2: Efficiency and effectiveness comparison of different prompting methods using GPT-4o as backend LLM. Left: Average accuracy across eight mathematical reasoning datasets over iterations (samples). Right: Average accuracy across eight mathematical reasoning datasets for different token consumption.
• Figure 3: (a) First, we prompt the LLM to generate a multi-step reasoning path for the given question. Next, we employ an iterative verify-then-revise process to progressively revise steps that are identified as incorrect by PSV, ultimately generating an error-free reasoning path. (b) We construct a process supervision dataset to fine-tune Llama-3-8B to obtain PSV. At each step, we use two different demonstrations to instruct the LLM in generating two subsequent steps, respectively. We define the quality of a step as its frequency in achieving the correct answer.
• Figure 4: Analysis of answer changes after five correction rounds. Correct $\rightarrow$ Correct: The answer remains correct; Incorrect $\rightarrow$ Correct: An incorrect answer is revised to a correct one; Correct $\rightarrow$ Incorrect: A correct answer is changed to an incorrect one; Incorrect $\rightarrow$ Incorrect: An incorrect answer is altered but remains incorrect. Use GPT-3.5-Turbo as the backend LLM.
• Figure 5: Performance comparison of StepCo and baselines using Llama-3-8B as the backend LLM. Compared to Best-of-10, StepCo achieves higher accuracy while consuming fewer tokens.