Not All LLM Reasoners Are Created Equal

TL;DR

This paper challenges the notion that LLMs have mastered grade-school math by revealing a systematic compositional reasoning gap between GSM8K and two-hop GSM problems. It introduces Compositional GSM, where the answer to $Q_2$ depends on $Q_1$'s solution, and conducts broad experiments across model families to quantify the gap with $\Delta = S_{ ext{comp}} - S_1 \cdot S_2$. The results show that cost-efficient and math-specialized LLMs suffer the largest gaps, while instruction-tuning effects vary with model size and finetuning can cause task overfitting; code generation helps smaller models but is not a universal fix. The study argues that leakage is not the primary cause of the gap and emphasizes distraction and poor second-hop reasoning as core limitations, advocating for more robust, out-of-distribution evaluations to accurately assess reasoning capabilities. Overall, the work highlights a mismatch between benchmark performance and genuine compositional reasoning, with practical implications for deploying LLMs in multi-hop reasoning tasks.

Abstract

We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs. To this end, we evaluate their performance on pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. Our findings reveal a significant reasoning gap in most LLMs, that is performance difference between solving the compositional pairs and solving each question independently. This gap is more pronounced in smaller, more cost-efficient, and math-specialized models. Moreover, instruction-tuning recipes and code generation have varying effects across LLM sizes, while finetuning on GSM can lead to task overfitting. Our analysis indicates that large reasoning gaps are not because of test-set leakage, but due to distraction from additional context and poor second-hop reasoning. Overall, LLMs exhibit systematic differences in their reasoning abilities, despite what their performance on standard benchmarks indicates.

Figures (13)

• Figure 1: Reasoning Gap: Most models demonstrate a noticeable gap between their reasoning performance on GSM8K and compositional GSM, in which pairs of GSM8K test questions are chained together so that the answer of the first question ($Q_1$) is a variable in the second one ($Q_2$). The model is required to correctly answer both questions to solve the problem. If a model has an accuracy of $S_1$ on the $Q_1$ set, and $S_2$ on $Q_2$ set, then the expected compositional GSM accuracy is ${S_1 \times S_2}$. The x-axis corresponds to the geometric mean $\sqrt{S_1 \times S_2}$, labeled GSM8K accuracy for simplicity. The trend-line $y=x^2$ is the expected compositional GSM accuracy.
• Figure 2: Example Problem from the Compositional GSM test. The answer of Question-1 ($\mathbf{Q_1}$) is a variable X in Question-2 ($\mathbf{Q_2}$). The model has to be able to solve the first question correctly in order to solve the second question. The new final answer of $\mathbf{Q_2}$ is calculated by modifying its code-form solution and executing it. We used a modified version of the code-form solutions from pal. Question-1 and the number to modify in Question-2 are chosen to have a new final answer which is a positive integer not too far from the old answer of Question-2.
• Figure 3: Reasoning Gap of notable open-weights and closed-source LLMs. Smaller, more cost-efficient and math specialized models have a bigger gap. See \ref{['fig:trend_main']} for GSM and compositional GSM accuracy.
• Figure 4: Cost efficient LLMs reason differently: showing four family of models, each having a high-cost and low-cost option. The numbers above the bars represents the reasoning gap defined in Eq \ref{['eq:comp_gap']}. Although the cheaper models perform similarly on the original GSM8K test, they show a significant decline in performance on the compositional GSM test.
• Figure 5: Impact of Instruction-Tuning on Compositional GSM. We compare pretrained and instruction-following tuned variant of models from Mistral, LLAMA3 and Gemma2 families. Numbers above bars represent improvements from instruction-tuning on each set. For smaller models (top), we observe that instruction-tuning results in substantial improvements on the original GSM8K test set, but a much smaller improvement on the compositional GSM test. However, this pattern does not typically hold for larger models (bottom).