Value-Aware Numerical Representations for Transformer Language Models

TL;DR

This work tackles the fragility of numerical understanding in Transformer language models by introducing a value-aware numerical encoding that prepends a dedicated <num> prefix to numeric inputs. The embedding of <num> is conditioned on the underlying value via a learnable function, and a projection is used during inference to align the generated numeric tokens with the value-based representation. Training combines three objectives to ensure consistency between training and inference while encouraging the model to internalize numeric magnitude. Evaluations on the NUPA benchmark show consistent improvements over a standard Transformer and a prior magnitude-aware baseline across numeric formats and operand lengths, demonstrating that explicit value encoding enhances fundamental numerical robustness with minimal architectural changes.

Abstract

Transformer-based language models often achieve strong results on mathematical reasoning benchmarks while remaining fragile on basic numerical understanding and arithmetic operations. A central limitation is that numbers are processed as symbolic tokens whose embeddings do not explicitly encode numerical value, leading to systematic errors. We introduce a value-aware numerical representation that augments standard tokenized inputs with a dedicated prefix token whose embedding is explicitly conditioned on the underlying numerical value. This mechanism injects magnitude information directly into the model's input space while remaining compatible with existing tokenizers and decoder-only Transformer architectures. Evaluation on arithmetic tasks shows that the proposed approach outperforms baselines across numerical formats, tasks, and operand lengths. These results indicate that explicitly encoding numerical value is an effective and efficient way to improve fundamental numerical robustness in language models.

Figures (3)

• Figure 1: Standard LLMs process numbers as symbolic tokens, which can lead to incorrect surface-form arithmetics. Our value-aware <num> token explicitly encodes numerical magnitude, enabling the model to reason over numbers as continuous measurements.
• Figure 2: Training and inference overview of the proposed approach. A standard Transformer is augmented with a special <num> token inserted before surface-form number tokens to explicitly encode numerical value. During training (left), the true numeric value $x$ is used by a trainable embedding module to compute the <num> embedding as $f(x)$, and a projection head is trained to align the hidden state with this representation, enforcing $proj(h_1) \approx f(x)$. During inference (right), when $x$ is unavailable, the projected hidden state $proj(h_1)$ replaces $f(x)$ and conditions the generation of numeric tokens.
• Figure 3: <num> embedding module variants. The MLP-based encoder represents integer and fractional parts as fixed-length decimal vectors projected through a feed-forward layer, while the RNN-based encoder processes variable-length digit sequences using separate GRUs for integer and fractional components. Auxiliary features (i.e., sign, digit count, decimal length) are concatenated in both cases.