Efficient numeracy in language models through single-token number embeddings

TL;DR

This paper addresses the numeracy limitations of current LLMs, where arithmetic often requires extensive reasoning or external tools. It introduces nine desiderata for single-token number encodings and presents BitTokens, a deterministic, IEEE 754-based bitwise encoding that maps numbers to a single token and enables learnable arithmetic. Empirical results show BitTokens outperform existing single-token and baseline strategies across a suite of numeracy tasks, even enabling small models to learn arithmetic with near-perfect accuracy and reducing the token burden per number. The approach has practical implications for extending problem length and complexity in technical domains, while inviting further validation on larger pretraining, production integration, and synergy with reasoning-based methods.

Abstract

To drive progress in science and engineering, large language models (LLMs) must be able to process large amounts of numerical data and solve long calculations efficiently. This is currently only possible through the use of external tools or extensive reasoning chains, either limiting the numerical intuition of LLMs or limiting the length of problems they can solve. We show that frontier LLMs require excessive amounts of reasoning tokens to solve even basic calculations, which is exacerbated by their tokenization strategies that split single numbers into multiple tokens. This motivates the need for efficient and effective single-token number encodings. We introduce a set of desiderata for such encodings and show that existing approaches fail to fulfill them. To address these shortcomings, we propose BitTokens, a novel tokenization strategy that embeds any number into a single token using its IEEE 754 binary floating-point representation. Through extensive experiments we show that our BitTokens allow even small language models to learn algorithms that solve basic arithmetic operations nearly perfectly. This newly gained efficiency could expand the length and complexity of problems language models can solve.

Figures (8)

• Figure 1: LLMs perform poorly on arithmetic tasks, requiring excessive reasoning tokens to achieve good performance. Our BitTokens tokenization strategy allows language models to solve arithmetic tasks both effectively and efficiently.
• Figure 2: While simple tasks such as addition and comparing numbers are almost perfectly solved by frontier LLMs, other tasks such as multiplication, division, calculating the standard deviation, or exponentiation remain difficult and require extensive reasoning to solve.
• Figure 3: Difficult numeracy tasks such as multiplication, division, exponentiation, and standard deviation can only be solved by frontier models using excessive reasoning tokens.
• Figure 4: While single digit is the superior multi-token strategy, our BitTokens outperforms it as well as all other single-token strategies on all 7 tasks in the multi-task setting.
• Figure 5: The magnitude distribution of addition pairs in our dataset. Operands with similar exponents are oversampled to increase difficulty.

Theorems & Definitions (5)

• Definition 4.1: sinusoidal encoding
• Lemma 4.2: additive homomorphism
• proof
• Proposition 4.3: non-locality and computational complexity of multiplication
• proof