Language Models Encode Numbers Using Digit Representations in Base 10

TL;DR

The paper interrogates why large language models struggle with simple numerical tasks and finds that they encode numbers as per-digit circular representations in base $10$, rather than as single, continuous numeric values. Through digit-wise probing and causal digit-interventions, it shows high accuracy in recovering individual digits and demonstrated that altering a digit can update the overall number in a predictable way, while direct value reconstruction remains unreliable. This digit-centric encoding explains error patterns observed in arithmetic tasks and provides a mechanistic lens for numerical reasoning in LLMs, with implications for robustness and interpretability across models and tokenizations. The work highlights a novel, digit-level mechanism that could guide future analyses and mitigation strategies for numerical reasoning in neural language systems, including extensions to word-form numbers and fractions.

Abstract

Large language models (LLMs) frequently make errors when handling even simple numerical problems, such as comparing two small numbers. A natural hypothesis is that these errors stem from how LLMs represent numbers, and specifically, whether their representations of numbers capture their numeric values. We tackle this question from the observation that LLM errors on numerical tasks are often distributed across the digits of the answer rather than normally around its numeric value. Through a series of probing experiments and causal interventions, we show that LLMs internally represent numbers with individual circular representations per-digit in base 10. This digit-wise representation, as opposed to a value representation, sheds light on the error patterns of models on tasks involving numerical reasoning and could serve as a basis for future studies on analyzing numerical mechanisms in LLMs.

Figures (7)

• Figure 1: An illustration of our key findings, suggesting that LLMs represent numbers on a per-digit base-10 basis: (a) on simple numerical tasks, LLMs often make errors that are close to the answer in 'digit space' rather than in value space, (b) though probing the exact number is hard, digit values can be decoded accurately.
• Figure 2: Error distribution in 7 operand addition.
• Figure 3: An illustration of our intervention on number representations via circular per-digit probes in base 10.
• Figure 4: Error distribution in 15 operand addition for GPT-4o.
• Figure 5: Visualization of the hidden states for natural number tokens (0 to 999) in layer 2 of Llama 3 8B, projected onto their top two principal components.