Language Models Encode the Value of Numbers Linearly

TL;DR

Experimental results support the existence of encoded number values in LLMs on different layers, and these values can be extracted via linear probes, proving the causal connection between encoded numbers and language model outputs.

Abstract

Large language models (LLMs) have exhibited impressive competence in various tasks, but their internal mechanisms on mathematical problems are still under-explored. In this paper, we study a fundamental question: how language models encode the value of numbers, a basic element in math. To study the question, we construct a synthetic dataset comprising addition problems and utilize linear probes to read out input numbers from the hidden states. Experimental results support the existence of encoded number values in LLMs on different layers, and these values can be extracted via linear probes. Further experiments show that LLMs store their calculation results in a similar manner, and we can intervene the output via simple vector additions, proving the causal connection between encoded numbers and language model outputs. Our research provides evidence that LLMs encode the value of numbers linearly, offering insights for better exploring, designing, and utilizing numeric information in LLMs.

Figures (18)

• Figure 1: Encoded number values in the hidden state of language models. We find that both the value of input numbers (blue and green) and calculation results (red) can be read out from the hidden state of language models via linear probes.
• Figure 2: Pearson coefficient ($\rho$) and out-of-sample $R^2$ of probes on different layers. $a$ and $b$ refer to the two input numbers denoted in Section \ref{['sec:dataset']}, and $o$ refers to the prediction of language models respectively. High $\rho$ and $R^2$ indicate the existence of encoded number values in the hidden states.
• Figure 3: Approximate accuracy (AAcc) and mean square error (MSE) of probes on different layers. $a$ and $b$ refer to the two input numbers denoted in Section \ref{['sec:dataset']}, and $o$ refers to the prediction of language models respectively. High AAcc and low MSE indicate precise number encoding.
• Figure 4: How the pattern of probe predictions on the first input number $a$ changes as the layer gets deeper. Probe predictions on different layers of LLaMA-2-7B show different patterns.
• Figure 5: The mean square error (MSE) of probes at different token positions on LLaMA-2-7B. <n1> represents the last token of the first input number $a$, and <n2> represents the last token of the second input number $b$, respectively. The rectangular pattern indicates that the value of an input number can be read out at any subsequent position.