Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristics

TL;DR

The paper investigates whether LLMs solve arithmetic through robust algorithms or memorization, using causal circuit analysis to identify a sparse, neuron-level arithmetic circuit. It reveals that a small set of MLP neurons implement diverse, input-pattern-based heuristics that, in combination, produce correct answers—a 'bag of heuristics' mechanism that emerges early in training and persists across models. Through targeted ablations and neuron-level probing, the work shows this mechanism is causally linked to arithmetic performance and explains its limitations and failure modes. The findings challenge the view of arithmetic reasoning as algorithmic or memorized and point to fundamental training-architecture implications for improving mathematical capabilities.

Abstract

Do large language models (LLMs) solve reasoning tasks by learning robust generalizable algorithms, or do they memorize training data? To investigate this question, we use arithmetic reasoning as a representative task. Using causal analysis, we identify a subset of the model (a circuit) that explains most of the model's behavior for basic arithmetic logic and examine its functionality. By zooming in on the level of individual circuit neurons, we discover a sparse set of important neurons that implement simple heuristics. Each heuristic identifies a numerical input pattern and outputs corresponding answers. We hypothesize that the combination of these heuristic neurons is the mechanism used to produce correct arithmetic answers. To test this, we categorize each neuron into several heuristic types-such as neurons that activate when an operand falls within a certain range-and find that the unordered combination of these heuristic types is the mechanism that explains most of the model's accuracy on arithmetic prompts. Finally, we demonstrate that this mechanism appears as the main source of arithmetic accuracy early in training. Overall, our experimental results across several LLMs show that LLMs perform arithmetic using neither robust algorithms nor memorization; rather, they rely on a "bag of heuristics".

Figures (18)

• Figure 1: Bag of heuristics visualization. We show that transformer LLMs solve arithmetic prompts by combining several unrelated heuristics, each activating according to rules based on the input values of operands, and boosting the logits of corresponding result tokens. These heuristics are manifested in single MLP neurons in mid to late layers.
• Figure 2: Llama3-8B arithmetic circuit discovery results. (a): Few attention heads have a high effect on arithmetic prompts. Most MLPs take part in the computation. The first MLP noticeably affects operand and operator positions, while mid- and late-layer MLPs influence the final position. (b): The arithmetic circuit in Llama3-8B. The attention heads project token information to the last position, where the middle- and late-layer MLPs promote the logits for the correct answer.
• Figure 3: Answer token probe accuracy. The linear probes are successful in extracting the correct answer from the final position, starting at layer $16$.
• Figure 4: Analyzing effect of individual circuit MLP neurons. Our results demonstrate that a small amount of neurons is required to correctly predict the result.
• Figure 5: Heuristic pattern examples. Each heatmap is the activation pattern of an example neuron, implementing a specific heuristic type. Within the heatmap, each pixel at location ($op_1, op_2$) represents the activation strength of the neuron under the addition prompt "$op_1+op_2=$".