Seeking Patterns, Not just Memorizing Procedures: Contrastive Learning for Solving Math Word Problems

TL;DR

This work investigates why math word problem solvers often memorize procedures rather than learn patterns. It reveals that semantic encoders cluster problems by prototype equations in higher layers and proposes a contrastive learning framework that pulls together problems with the same prototype while pushing apart different patterns, using contrastive triples derived from equation trees. The model is trained with a combined loss that balances equation generation and pattern-aware contrastive objectives, evaluated on Math23k and MathQA in monolingual and multilingual settings, achieving consistent improvements. The findings highlight a path toward language-agnostic pattern understanding in MWPs and demonstrate practical gains across languages, with evidence that pattern-centered representations reduce incorrect solutions.

Abstract

Math Word Problem (MWP) solving needs to discover the quantitative relationships over natural language narratives. Recent work shows that existing models memorize procedures from context and rely on shallow heuristics to solve MWPs. In this paper, we look at this issue and argue that the cause is a lack of overall understanding of MWP patterns. We first investigate how a neural network understands patterns only from semantics, and observe that, if the prototype equations are the same, most problems get closer representations and those representations apart from them or close to other prototypes tend to produce wrong solutions. Inspired by it, we propose a contrastive learning approach, where the neural network perceives the divergence of patterns. We collect contrastive examples by converting the prototype equation into a tree and seeking similar tree structures. The solving model is trained with an auxiliary objective on the collected examples, resulting in the representations of problems with similar prototypes being pulled closer. We conduct experiments on the Chinese dataset Math23k and the English dataset MathQA. Our method greatly improves the performance in monolingual and multilingual settings.

Figures (8)

• Figure 1: The visualization of the problem representations by T-SNE. "Prob." and "Eq" are short for the math word problem and its solution equation. The problem A and B are in the same prototype equation "$n_1-n_2$". The problem C and D are semantically similar.
• Figure 2: The T-SNE visualization of problem representations in different epochs and different layers. Different colors represent different prototype equations. The model achieves the highest accuracy at the training epoch 43.
• Figure 3: Similarities of problem representations in different BERT layers. The blue polyline corresponds to the semantically similar problems. The red polyline corresponds to problems with same prototype equation.
• Figure 4: Model performance in each distance interval. The interval index $x$ indicates the cosine distances are in the interval $[0.1 \times (x-1) , 0.1 \times x)$. The dotted line is computed by polynomial least squares fitting.
• Figure 5: An overview of our model.