Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search

Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Conghui He, Zenan Zhou, Wentao Zhang

TL;DR

BEATS tackles the persistent challenge of mathematical problem solving in LLMs by addressing (i) ambiguous prompts, (ii) high inference cost from expansive search, and (iii) unreliable verification. It introduces disambiguation-focused prompts, a pruning tree search to bound inference, and a back-verification step that rechecks candidate answers against the original problem, yielding strong results on MATH, GSM8K, SVAMP, SimulEq, and NumGLUE. Empirical findings show BEATS substantially improves baseline performance even without fine-tuning, with notable gains such as achieving 61.52% on MATH with Qwen2-7B-Instruct and 83.02% on GSM8K, surpassing some closed-domain baselines. The approach demonstrates robust, scalable enhancements to mathematical reasoning that reduce computation while maintaining high accuracy, offering a practical path toward broader deployment in complex problem domains.

Abstract

Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. However, they still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. Previous studies have employed techniques such as supervised fine-tuning (SFT), prompt engineering, and search-based methods to improve the mathematical problem-solving abilities of LLMs. Despite these efforts, their performance remains suboptimal and demands substantial computational resources. To address this issue, we propose a novel approach, BEATS, to enhance mathematical problem-solving abilities. Our method leverages newly designed prompts that guide the model to iteratively rewrite, advance by one step, and generate answers based on previous steps. Additionally, we introduce a new back-verification technique that uses LLMs to validate the correctness of the generated answers. Furthermore, we employ a pruning tree search to optimize search time while achieving strong performance. Notably, our method improves Qwen2-7b-Instruct's score from 36.94 to 61.52, outperforming GPT4's 42.5 on the MATH benchmark.

BEATS: Optimizing LLM Mathematical Capabilities with BackVerify and Adaptive Disambiguate based Efficient Tree Search

TL;DR

BEATS tackles the persistent challenge of mathematical problem solving in LLMs by addressing (i) ambiguous prompts, (ii) high inference cost from expansive search, and (iii) unreliable verification. It introduces disambiguation-focused prompts, a pruning tree search to bound inference, and a back-verification step that rechecks candidate answers against the original problem, yielding strong results on MATH, GSM8K, SVAMP, SimulEq, and NumGLUE. Empirical findings show BEATS substantially improves baseline performance even without fine-tuning, with notable gains such as achieving 61.52% on MATH with Qwen2-7B-Instruct and 83.02% on GSM8K, surpassing some closed-domain baselines. The approach demonstrates robust, scalable enhancements to mathematical reasoning that reduce computation while maintaining high accuracy, offering a practical path toward broader deployment in complex problem domains.

Abstract

Large Language Models (LLMs) have exhibited exceptional performance across a broad range of tasks and domains. However, they still encounter difficulties in solving mathematical problems due to the rigorous and logical nature of mathematics. Previous studies have employed techniques such as supervised fine-tuning (SFT), prompt engineering, and search-based methods to improve the mathematical problem-solving abilities of LLMs. Despite these efforts, their performance remains suboptimal and demands substantial computational resources. To address this issue, we propose a novel approach, BEATS, to enhance mathematical problem-solving abilities. Our method leverages newly designed prompts that guide the model to iteratively rewrite, advance by one step, and generate answers based on previous steps. Additionally, we introduce a new back-verification technique that uses LLMs to validate the correctness of the generated answers. Furthermore, we employ a pruning tree search to optimize search time while achieving strong performance. Notably, our method improves Qwen2-7b-Instruct's score from 36.94 to 61.52, outperforming GPT4's 42.5 on the MATH benchmark.
Paper Structure (29 sections, 3 equations, 7 figures, 3 tables, 1 algorithm)

This paper contains 29 sections, 3 equations, 7 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Suboptimal Prompts
  3. High Computational Cost
  4. Ineffective Verification Method
  5. Related Work
  6. Math Large Language Models
  7. Prompt Engineering for Large Language Models
  8. Reasoning in Large Language Models
  9. Method
  10. Prompt Design
  11. One Step Forward
  12. Giving the Final Answer
  13. Disambiguation
  14. Pruning Tree Search
  15. Back-Verification
  16. ...and 14 more sections

Figures (7)

  • Figure 1: We provide a straightforward example to illustrate our BEATS method. First, we construct a tree search using three distinct actions. Next, we apply back verification to achieve the correct answer.
  • Figure 2: Visualization of the search algorithm in BEATS.
  • Figure 3: From this figure, we observe that models are more likely to deduce errors when using majority voting but can achieve the correct answer through back verification.
  • Figure 4: From this figure, we observe that some questions may contain ambiguity, which can be resolved by using the disambiguation operation to generate a clarified version of the question.
  • Figure 5: Average tokens needed for solving different problems.
  • ...and 2 more figures