Number Representations in LLMs: A Computational Parallel to Human Perception

TL;DR

This work investigates whether numerical magnitudes in LLMs are encoded along a non-uniform, logarithmic-like number line, mirroring human perception. It introduces a framework using $T: \mathbb{R}^d \to \mathbb{R}^p$ projections via PCA/PLS and metrics like Spearman $\rho$ and Scaling Rate Index $\beta$ to quantify monotonicity and sublinear spacing in $f_{ ext{LLM}}$. Empirical results show that PCA captures systematic sublinearity in numerical embeddings across models, while PLS yields higher monotonicity and fit but can obscure intrinsic geometric structure. Real-world data experiments with birth years and populations demonstrate model-specific monotonicity and compression, supporting a structured but non-uniform internal numeracy in LLMs with implications for interpretable numerical reasoning.

Abstract

Humans are believed to perceive numbers on a logarithmic mental number line, where smaller values are represented with greater resolution than larger ones. This cognitive bias, supported by neuroscience and behavioral studies, suggests that numerical magnitudes are processed in a sublinear fashion rather than on a uniform linear scale. Inspired by this hypothesis, we investigate whether large language models (LLMs) exhibit a similar logarithmic-like structure in their internal numerical representations. By analyzing how numerical values are encoded across different layers of LLMs, we apply dimensionality reduction techniques such as PCA and PLS followed by geometric regression to uncover latent structures in the learned embeddings. Our findings reveal that the model's numerical representations exhibit sublinear spacing, with distances between values aligning with a logarithmic scale. This suggests that LLMs, much like humans, may encode numbers in a compressed, non-uniform manner.

Figures (16)

• Figure 1: Logarthmic mental number line hypothesis asserts that humans innately percieve numbers on a logarithmic scale. Image source fritz2013development.
• Figure 2: The overall graphical representation of our method. Numbers are passed to the model in form of a prompt and the internal representations are captured from the embeddings corresponding to token '='. At every layer, we perform PCA projections onto one and two dimensional subspaces and pick a layer with highest explained variance ($\sigma^2$) score to further analyze monotonicity and scaling of number representations.
• Figure 3: Layer-wise analysis of four models on numerical groups, showing explained variance ($\sigma^2$), monotonicity ($\rho$), and Scaling Rate Index ($\beta$). The layer with maximum $\sigma^2$ aligns with peak $\rho$, indicating optimal numerical encoding.
• Figure 4: Projections of numerical representations (y-axis) against their log-scaled magnitudes (x-axis) for the layer with the highest explained variance in four models. Sublinearity and monotonicity ($\rho$) are indicated above each subfigure, demonstrating consistent sublinear trends and strong monotonic relationships across models.
• Figure 5: Projections of letters representations (y-axis) against their log-scaled magnitudes (x-axis) assigned proportional to their length, for the layer with the highest explained variance in four models. Sublinearity and monotonicity ($\rho$) are indicated above each subfigure, demonstrating consistent sublinear trends and strong monotonic relationships across models.