Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Expressive Power of Floating-Point Transformers

Sejun Park, Yeachan Park, Geonho Hwang

TL;DR

This work analyzes how finite-precision floating-point arithmetic affects the expressive power of self-attention transformers. It demonstrates that floating-point transformers are not generally permutation-equivariant and cannot universally approximate all permutation-equivariant sequence-to-sequence functions when the input length $n$ is large, due to FP-specific phenomena like non-associativity. However, when $n$ is bounded (for example, $n \le 6\cdot 2^p - 2$ for mantissa precision $p$), FP transformers can represent all permutation-equivariant FP functions, and the authors characterize a minimal equivariance structure $\pi_{(1,2)}^n$ and the detrimental effect of additive positional encodings. The results illuminate practical implications for transformer design on real hardware, highlighting inherent biases and the need for alternative positional-encoding strategies to preserve expressivity under finite-precision computation.

Abstract

The study on the expressive power of transformers shows that transformers are permutation equivariant, and they can approximate all permutation-equivariant continuous functions on a compact domain. However, these results are derived under real parameters and exact operations, while real implementations on computers can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we investigate the representability of floating-point transformers that use floating-point parameters and floating-point operations. Unlike existing results under exact operations, we first show that floating-point transformers can represent a class of non-permutation-equivariant functions even without positional encoding. Furthermore, we prove that floating-point transformers can represent all permutation-equivariant functions when the sequence length is bounded, but they cannot when the sequence length is large. We also found the minimal equivariance structure in floating-point transformers, and show that all non-trivial additive positional encoding can harm the representability of floating-point transformers.

On the Expressive Power of Floating-Point Transformers

TL;DR

This work analyzes how finite-precision floating-point arithmetic affects the expressive power of self-attention transformers. It demonstrates that floating-point transformers are not generally permutation-equivariant and cannot universally approximate all permutation-equivariant sequence-to-sequence functions when the input length is large, due to FP-specific phenomena like non-associativity. However, when is bounded (for example, for mantissa precision ), FP transformers can represent all permutation-equivariant FP functions, and the authors characterize a minimal equivariance structure and the detrimental effect of additive positional encodings. The results illuminate practical implications for transformer design on real hardware, highlighting inherent biases and the need for alternative positional-encoding strategies to preserve expressivity under finite-precision computation.

Abstract

The study on the expressive power of transformers shows that transformers are permutation equivariant, and they can approximate all permutation-equivariant continuous functions on a compact domain. However, these results are derived under real parameters and exact operations, while real implementations on computers can only use a finite set of numbers and inexact machine operations with round-off errors. In this work, we investigate the representability of floating-point transformers that use floating-point parameters and floating-point operations. Unlike existing results under exact operations, we first show that floating-point transformers can represent a class of non-permutation-equivariant functions even without positional encoding. Furthermore, we prove that floating-point transformers can represent all permutation-equivariant functions when the sequence length is bounded, but they cannot when the sequence length is large. We also found the minimal equivariance structure in floating-point transformers, and show that all non-trivial additive positional encoding can harm the representability of floating-point transformers.
Paper Structure (35 sections, 23 theorems, 81 equations)

This paper contains 35 sections, 23 theorems, 81 equations.

Table of Contents

  1. Introduction
  2. Contributions
  3. Organization
  4. Preliminaries
  5. Notations
  6. Floating-point arithmetic
  7. Floating-point transformer
  8. Main results
  9. Floating-point transformers are not $\mathcal{S}_n$-equivariant
  10. Floating-point transformers cannot represent all $\mathcal{S}_n$-equivariant functions
  11. Discussions
  12. Inductive biases in floating-point transformers
  13. Effects of positional encoding
  14. Proofs
  15. Proof of \ref{['thm:diagonal-approx']}
  16. ...and 20 more sections

Key Result

Theorem 1

Let $d_\text{in},d_\text{out},n\in\mathbb{N}$ such that $n\ge2$ and Then, for any ${\pi_{(1,2)}^n}$-equivariant $f^*:\Delta_n\to\mathbb{F}^{d_\text{out}\times n}$, there exists a floating-point transformer $f:\mathbb{F}^{d_\text{in}\times n}\to\mathbb{F}^{d_\text{out}\times n}$ such that $f=f^*$ on $\Delta_n$.

Theorems & Definitions (35)

  • Theorem 1
  • Definition 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Definition 2
  • Theorem 5
  • Lemma 6
  • Lemma 7
  • Lemma 8
  • ...and 25 more