Transformers Can Overcome the Curse of Dimensionality: A Theoretical Study from an Approximation Perspective

TL;DR

The paper investigates approximation of the Hölder class $\mathcal{H}_{Q}^{\beta}([0,1]^{d\times n},\mathbb{R}^{d\times n})$ by Transformers and shows, via a Kolmogorov–Arnol'd Representation Theorem (KST) based construction, that a Transformer with a single self-attention layer and a small feedforward network can overcome the curse of dimensionality. It translates the high-dimensional approximation into univariate outer-function approximation and then implements these via memorization-enabled FFN blocks and a self-attention step to sum across columns; the Cantor-set embedding and the approximant $\phi_K$ are used to realize the KST decomposition. The main results provide $d_\infty$ and $d_p$ approximation guarantees with explicit depth/width bounds that scale polylogarithmically with $1/\epsilon$ (and can be constant under broader activations), demonstrating the Transformer’s strong expressive power under weak assumptions. The approach avoids reliance on contextual mapping and offers a translation technique to leverage known FNN approximation results for Transformer architectures, with potential implications for both theory and practical sequence-modeling tasks.

Abstract

The Transformer model is widely used in various application areas of machine learning, such as natural language processing. This paper investigates the approximation of the Hölder continuous function class $\mathcal{H}_{Q}^β\left([0,1]^{d\times n},\mathbb{R}^{d\times n}\right)$ by Transformers and constructs several Transformers that can overcome the curse of dimensionality. These Transformers consist of one self-attention layer with one head and the softmax function as the activation function, along with several feedforward layers. For example, to achieve an approximation accuracy of $ε$, if the activation functions of the feedforward layers in the Transformer are ReLU and floor, only $\mathcal{O}\left(\log\frac{1}ε\right)$ layers of feedforward layers are needed, with widths of these layers not exceeding $\mathcal{O}\left(\frac{1}{ε^{2/β}}\log\frac{1}ε\right)$. If other activation functions are allowed in the feedforward layers, the width of the feedforward layers can be further reduced to a constant. These results demonstrate that Transformers have a strong expressive capability. The construction in this paper is based on the Kolmogorov-Arnold Representation Theorem and does not require the concept of contextual mapping, hence our proof is more intuitively clear compared to previous Transformer approximation works. Additionally, the translation technique proposed in this paper helps to apply the previous approximation results of feedforward neural networks to Transformer research.

Figures (1)

• Figure 1: The flowchart for computing $\phi_K(x)$ when $K = 4$.

Theorems & Definitions (28)

• Proposition 1: KST
• Definition 1: Hölder continuous function
• Definition 2
• Proposition 2: schmidt2021kolmogorov, Theorem 2
• Theorem 1
• Theorem 2
• Remark 1
• Remark 2
• Remark 3
• Remark 4