Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers

TL;DR

The paper addresses how low-precision quantization affects Transformer expressivity, focusing on equality-like computations as a benchmark. It develops a fine-grained, tight tradeoff by pairing explicit finite-precision Transformer constructions with one-way communication complexity lower bounds, proving that for every $p>0$ there exists a function $Γ$ (an EQ-inspired task) that a one-layer Transformer can compute with $p$ bits but not with $p-1$ bits, with analogous results for fixed-point and floating-point formats. The approach yields a sharp one-bit threshold and highlights the role of input encoding and padding in achieving optimal tradeoffs, offering insights into why equality-check tasks are particularly sensitive to quantization. The empirical experiments corroborate the theory, showing accuracy degradation under reduced precision and partial restoration via quantization-aware training, and provide practical guidance: quantization levels should be chosen in relation to the expected length of equality checks in the task domain, with potential extensions to multi-layer architectures as a future direction.

Abstract

Quantization reduces the numerical precision of Transformer computations and is widely used to accelerate inference, yet its effect on expressivity remains poorly characterized. We demonstrate a fine-grained theoretical tradeoff between expressivity and precision: For every p we exhibit a function Γ, inspired by the equality function, and prove that a one-layer softmax Transformer can compute Γ, with p bits of precision, but not with p-1 bits of precision. This result concretely explains the widely observed phenomenon of empirical loss of expressivity when quantization is used. Practically, it suggests that tasks requiring equality-like comparisons (exact match, membership, etc.) are especially sensitive to quantization. Dropping even one bit can cross a threshold where the model cannot represent the needed comparison reliably. Thus, it paves the way for developing heuristics that will help practitioners choose how much quantization is possible: the precision should be chosen as a function of the length of equality to be checked for the specific task. Our proofs combine explicit finite-precision Transformer constructions with communication-complexity lower bounds, yielding a tight "one-bit" threshold.

Figures (12)

• Figure 1: Learning curves with mean and one standard deviation for $m=100$.
• Figure 2: Learning curves with mean and one standard deviation for $m=50$.
• Figure 3: Learning curves with mean and one standard deviation for $m=30$.
• Figure 4: Learning curves with mean and one standard deviation for $m=15$.
• Figure 5: QAT curves for $m=100$ using fixed-point precision

Theorems & Definitions (45)

• Theorem 1.1: Informal
• Theorem 2.1: Tradeoff with Fixed-Point Precision for Equality
• Lemma 2.1: Fixed-Point Lower Bound
• Lemma 2.1: Fixed-Point Upper bound
• proof : Proof overview.
• Theorem 2.2: Tight Tradeoff with Fixed-Point Precision
• Lemma 2.2: Lower Bound
• Lemma 2.2: Upper Bound
• proof : Proof overview.
• proof : Proof of Theorem \ref{['thm:tradeoff']}