Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrieval

TL;DR

This paper presents how analogy from similarly structured questions can improve LLMs' problem-solving capabilities for MWPs, and relies on the retrieval of problems with similar computational graphs to the given question to serve as exemplars in the prompt, providing the correct reasoning path for the generation model to refer to.

Abstract

Large language models (LLMs) are known to struggle with complicated reasoning tasks such as math word problems (MWPs). In this paper, we present how analogy from similarly structured questions can improve LLMs' problem-solving capabilities for MWPs. Specifically, we rely on the retrieval of problems with similar computational graphs to the given question to serve as exemplars in the prompt, providing the correct reasoning path for the generation model to refer to. Empirical results across six math word problem datasets demonstrate the effectiveness of our proposed method, which achieves a significant improvement of up to 6.7 percent on average in absolute value, compared to baseline methods. These results highlight our method's potential in addressing the reasoning challenges in current LLMs.

Figures (6)

• Figure 1: An example of a math word problem with its computational graph.
• Figure 2: Flowchart of Retriver Training. This figure illustrates the process of training a retriever model (encoder) with contrastive learning to identify structurally similar math word problems. Each question is encoded into an embedding based on its text. Positive pairs are formed by pairing examples with matching computational graph structures, while in-batch negatives serve as contrasting examples with different structures.
• Figure 3: Case study on the retrieved data with our model and BGE respectively. The retrieved data using trained retriever have similar computational graphs with the query question, while the computational graphs are different for retrieved data using BGE model.
• Figure 4: The scatter plot of our trained retriever (left) and BGE (right) on 100 random samples from Math23k. There is a stronger positive correlation between computational graph similarity and embedding similarity for data with trained retriever than the BGE model.
• Figure 5: The relationship between model performance on aqua_rat dataset and the amount of data used. The performance stops to boost when using more than 25% of training data.