Arithmetic with Language Models: from Memorization to Computation

TL;DR

This work investigates how a small, non-pretrained Transformer LM can perform binary arithmetic beyond its training data, framing the computation as an Encoding-Regression-Decoding (ERD) process. Through controlled experiments on binary addition and multiplication, the authors show the model learns these tasks with strong generalization, and provide evidence that computation occurs as regression in a learned value space after an initial encoding and before a final decoding step. The study employs interpolation/extrapolation analyses, internal representation correlations, and amnesic probing to support the ERD account, and demonstrates the feasibility of numeric regression as a core mechanism in language models. The findings illuminate how LMs process numbers and suggest a generalizable pathway for integrating arithmetic and reasoning capabilities into practical NLP systems.

Abstract

A better understanding of the emergent computation and problem-solving capabilities of recent large language models is of paramount importance to further improve them and broaden their applicability. This work investigates how a language model, trained to predict the next token, can perform arithmetic computations generalizing beyond training data. Binary addition and multiplication constitute a good testbed for this purpose, since they require a very small vocabulary and exhibit relevant input/output discontinuities making smooth input interpolation ineffective for novel data. We successfully trained a light language model to learn these tasks and ran a number of experiments to investigate the extrapolation capabilities and internal information processing. Our findings support the hypothesis that the language model works as an Encoding-Regression-Decoding machine where the computation takes place in the value space once the input token representation is mapped to an appropriate internal representation.

Figures (8)

• Figure 1: Sequence accuracy. From the left: addition and multiplication. Results are averaged over five runs. Note that, training and validation curves are almost overlapped. At the end of training the Mean Absolute Error (MAE) on the validation set, between the real and generated operation results, is 0 and 1.3 for addition and multiplication, respectively.
• Figure 2: Sequence accuracy using random output in the training set. Results are averaged over five runs.
• Figure 3: Sequence accuracy on Random, $VS_t$, and $VS_v$ validation subsets for addition (left) and multiplication (right). Results are averaged over five runs. $VS_t$ reaches 100% accuracy on additions (the same of Random split) and 97.5% accuracy on multiplication (just 1.4% less than random split); $VS_v$ reaches 93.7% on addition and 94.3% on multiplication (6.3% and 4.6% less than Random split, respectively).
• Figure 4: Pearson correlation between ordered sets of distances for addition (a) and multiplication (b). Each cell denotes the correlation between the two ordered set of distances specified in the corresponding row and column. Note that since for addition in this experiment the output value is always twice the input, the correlation values (blue and green cells) are the same for $d_{in,\_}$ and $d_{out,\_}$ block of values. Graphs (c) and (d) show the correlations of output distances $d_{out,t}$ (at token level - blue curves) and $d_{out,v}$ (at value level - orange curves) with the embedding distances $d_{dec_i}$ across the 6 decoder layers for addition and multiplication, respectively.
• Figure 5: Example of 4-digit binary multiplication. The sum can be performed incrementally with a two-operand adder.