Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics

Ting-Ruen Wei, Haowei Liu, Xuyang Wu, Yi Fang

TL;DR

This survey analyzes how feedback-based multi-step reasoning enhances mathematical problem solving in large language models, distinguishing training-based (step-level PRMs and ORM-based supervision) from training-free approaches (self-evaluation, logits, and external-tool verification) and detailing their roles in aggregation, search, and refinement. It presents a taxonomy of methods that rely on step-level versus outcome-level guidance, including hybrid strategies, and surveys datasets and evaluation challenges relevant to math reasoning. Key contributions include a comprehensive mapping of reward-modeling techniques (PRMs/ORMs), search strategies (MCTS, beam search), and prompting/refinement methods, along with a critical discussion of reward hacking and distribution shift. The work aims to establish foundational benchmarks and guidance to advance robust, efficient, and scalable math reasoning with LLMs across diverse problem domains and languages.

Abstract

Recent progress in large language models (LLM) found chain-of-thought prompting strategies to improve the reasoning ability of LLMs by encouraging problem solving through multiple steps. Therefore, subsequent research aimed to integrate the multi-step reasoning process into the LLM itself through process rewards as feedback and achieved improvements over prompting strategies. Due to the cost of step-level annotation, some turn to outcome rewards as feedback. Aside from these training-based approaches, training-free techniques leverage frozen LLMs or external tools for feedback at each step to enhance the reasoning process. With the abundance of work in mathematics due to its logical nature, we present a survey of strategies utilizing feedback at the step and outcome levels to enhance multi-step math reasoning for LLMs. As multi-step reasoning emerges a crucial component in scaling LLMs, we hope to establish its foundation for easier understanding and empower further research.

A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematics

TL;DR

This survey analyzes how feedback-based multi-step reasoning enhances mathematical problem solving in large language models, distinguishing training-based (step-level PRMs and ORM-based supervision) from training-free approaches (self-evaluation, logits, and external-tool verification) and detailing their roles in aggregation, search, and refinement. It presents a taxonomy of methods that rely on step-level versus outcome-level guidance, including hybrid strategies, and surveys datasets and evaluation challenges relevant to math reasoning. Key contributions include a comprehensive mapping of reward-modeling techniques (PRMs/ORMs), search strategies (MCTS, beam search), and prompting/refinement methods, along with a critical discussion of reward hacking and distribution shift. The work aims to establish foundational benchmarks and guidance to advance robust, efficient, and scalable math reasoning with LLMs across diverse problem domains and languages.

Abstract

Recent progress in large language models (LLM) found chain-of-thought prompting strategies to improve the reasoning ability of LLMs by encouraging problem solving through multiple steps. Therefore, subsequent research aimed to integrate the multi-step reasoning process into the LLM itself through process rewards as feedback and achieved improvements over prompting strategies. Due to the cost of step-level annotation, some turn to outcome rewards as feedback. Aside from these training-based approaches, training-free techniques leverage frozen LLMs or external tools for feedback at each step to enhance the reasoning process. With the abundance of work in mathematics due to its logical nature, we present a survey of strategies utilizing feedback at the step and outcome levels to enhance multi-step math reasoning for LLMs. As multi-step reasoning emerges a crucial component in scaling LLMs, we hope to establish its foundation for easier understanding and empower further research.

Paper Structure

This paper contains 25 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Problem Statement
  4. Preliminaries
  5. Training-based Strategies
  6. Step-level Feedback
  7. Aggregation
  8. Search
  9. Refinement
  10. Outcome-level Feedback
  11. Discriminative ORM
  12. Generative ORM
  13. Rule-based Rewards
  14. Step- and Outcome-level Feedback
  15. Training-free Approaches
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Our survey coverage on feedback-based multi-step reasoning. We focus on strategies that utilize feedback for enhancing the LLM performance across multiple steps on math problems.
  • Figure 2: Illustrative example. Outcome reward evaluates the outcome (GSM8K dataset cobbe2021training), while process reward evaluates each reasoning step (Math-Shepherd dataset wang2024math).
  • Figure 3: Taxonomy of multi-step math reasoning through feedback. Training-based approaches leverage step- or outcome-level feedback. Step-level feedback can be aggregated for voting or best-of-N, or assist search or refinement. Outcome-level feedback can train discriminative or generative ORMs, or be determined by rules. Training-free techniques evaluate each step by direct LLM responses, LLM token logits, or an external tool to obtain step-level feedback.