Fill in the Blank: Exploring and Enhancing LLM Capabilities for Backward Reasoning in Math Word Problems

TL;DR

The paper addresses backward reasoning in math word problems, a less-explored counterpart to forward problem solving. It adapts three forward reasoning strategies—Rephrase, PAL-Tools, and Check Your Work—and introduces a Bayesian ensemble that leverages a forward verifier to boost accuracy on backward tasks. Extensive experiments on backward versions of GSM8k, SVAMP, and MultiArith demonstrate progressive performance gains, with the ensemble surpassing state-of-the-art forward strategies adapted to backward reasoning. The work also extends to phrase-masked backward tasks and analyzes the contributions of prompting, verification, and ensemble design, offering a viable path for abductive-style reasoning in LLMs. Overall, the approach yields meaningful improvements and provides a framework for applying backward reasoning to other domains.

Abstract

While forward reasoning (i.e., find the answer given the question) has been explored extensively in recent literature, backward reasoning is relatively unexplored. We examine the backward reasoning capabilities of LLMs on Math Word Problems (MWPs): given a mathematical question and its answer, with some details omitted from the question, can LLMs effectively retrieve the missing information? On modifying three benchmark datasets for this task, to evaluate this task: GSM8k, SVAMP, and MultiArith, we find a significant drop in the accuracy of models on this task compared to forward reasoning across SOTA LLMs (GPT4, GPT3.5, PaLM-2, and LLaMa). Motivated by the fact backward reasoning can be seen as the ''inverse'' of forward reasoning, we propose variations of three different forward reasoning strategies to improve performance. Rephrase reformulates the given problem into a forward reasoning problem, PAL-Tools combines the idea of Program-Aided LLMs to produce a set of equations that can be solved by an external solver, and Check your Work exploits the availability of natural verifier of high accuracy in the forward direction, interleaving solving and verification steps. Finally, realizing that each of our base methods correctly solves a different set of problems, we propose a novel Bayesian formulation for creating an ensemble over the base methods to further boost the accuracy. Extensive experimentation demonstrates successive improvement in the performance of LLMs on the backward reasoning task, using our strategies, with our ensemble-based method resulting in significant performance gains compared to the SOTA forward reasoning strategies we adapt.

Figures (19)

• Figure 1: A summary of the prompting techniques we adapt
• Figure 2: An illustrative example of how the ensembling of base models works together with a verifier.
• Figure 3: Overlap between problems in GSM8k$_\text{B}$ that different base techniques can solve
• Figure 4: Rephrasing with Linguistic prompt
• Figure 5: Rephrasing with Algebraic prompt