Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems

TL;DR

The paper addresses the bottleneck of semantic misunderstanding in chain-of-thought prompting for complex math word problems and proposes Deeply Understanding the Problems (DUP), a three-stage zero-shot prompting framework to extract the core question and the necessary problem-solving information before solving. DUP consistently improves over zero-shot CoT across arithmetic, commonsense, and symbolic reasoning benchmarks, achieving state-of-the-art GSM8K and SVAMP scores with GPT-4 (GSM8K 97.1%, SVAMP 94.2%). The authors provide extensive ablations, error analyses, and demonstrations of compatibility with self-consistency and open-source LLMs, along with a cost-efficient variant (DUP-s). The work presents a practical, plug-and-play approach that enhances math reasoning without additional training and suggests broad applicability to non-reasoning tasks as future work.

Abstract

Chain-of-Thought (CoT) prompting has enhanced the performance of Large Language Models (LLMs) across various reasoning tasks. However, CoT still falls short in dealing with complex math word problems, as it usually suffers from three pitfalls: semantic misunderstanding errors, calculation errors, and step-missing errors. Prior studies involve addressing the calculation errors and step-missing errors, but neglect the semantic misunderstanding errors, which is the major factor limiting the reasoning performance of LLMs. To this end, we propose a simple-yet-effective method, namely Deeply Understanding the Problems (DUP), to improve the LLMs' math problem-solving ability by addressing semantic misunderstanding errors. The core of our method is to encourage the LLMs to deeply understand the problems and extract the key problem-solving information used for better reasoning. Extensive experiments on 10 diverse reasoning benchmarks show that our DUP method consistently outperforms the other counterparts by a large margin. More encouragingly, DUP achieves a new SOTA result on the GSM8K benchmark, with an accuracy of 97.1% under the zero-shot setting.

Figures (7)

• Figure 1: Error analysis of GSM8K problems with incorrect answers returned by zero-shot CoT and our DUP using GPT-3.5 LLM. We randomly sample 300 GSM8K problems, and follow cot_wei and wang2023planandsolve to assign the "Semantic Misunderstanding", "Calculation Error" and "Step-missing Error" to each incorrect answer. The detailed prompts for error analysis are shown in Table \ref{['tab:prompt_error']}. We see that our DUP method effectively reduces the errors among all types.
• Figure 2: Illustration of our DUP prompting strategy, which contains three-stage processes: ❶ revealing the core question from the original input; ❷ extracting the problem-solving information based on the core question; ❸ generating and extracting the final answer via understanding the core question and problem-solving information.
• Figure 3: Results of DUP Prompting with and without self-consistency(SC) using GPT-3.5-turbo LLM on GSM8K and SVAMP.
• Figure 4: Analysis of different information extractors used in our DUP. We use the GPT-4, GPT-3.5-turbo, and Llama-2-Chat 70b to extract core question (Stage1) and problem-solving information (Stage2) extractor, and leverage the extracted contents to guide the responses of GPT-3.5-turbo (Stage3). We see that more accurate core questions and problem-solving information lead to better performance.
• Figure 5: Performance of DUP and DUP-s across various reasoning tasks on GPT-3.5-Turbo, where DUP-s merges the three-stage prompts into one prompt. Orange and Blue dashlines represent the average accuracy of DUP and DUP-s, respectively. We see that our simplified DUP-s also achieves remarkable performance with less inference budget.