Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation Rate of the Transformer Architecture for Sequence Modeling

Haotian Jiang, Qianxiao Li

TL;DR

This work develops Jackson-type approximation rate results for a simplified, single-layer Transformer with one head, linking the model’s approximation power to a low-rank temporal coupling in target functions. By decomposing targets into pairwise temporal interactions and element-wise components via a POD-based temporal coupling, and by defining suitable complexity measures ($C_0$, $C_1^{(\alpha)}$, $C_2^{(\beta)}$), the authors derive explicit error bounds showing how the rate improves as the attention dimension $m_h$ and feed-forward budget $m_{FF}$ grow. The representation theorem confirms that any target can be expressed in the proposed target form, and numerical demonstrations with synthetic targets and ViT on CIFAR-10 support the theoretical insights, including practical low-rank patterns in real data. A comparative analysis with RNNs reveals that Transformers can be robust to temporal ordering changes but can be sensitive to temporal mixing, implying complementary strengths between the architectures. Collectively, the results illuminate when Transformers excel in sequence modeling and guide future work toward multi-heads, deeper architectures, and input preprocessing to mitigate temporal mixing.

Abstract

The Transformer architecture is widely applied in sequence modeling applications, yet the theoretical understanding of its working principles remains limited. In this work, we investigate the approximation rate for single-layer Transformers with one head. We consider a class of non-linear relationships and identify a novel notion of complexity measures to establish an explicit Jackson-type approximation rate estimate for the Transformer. This rate reveals the structural properties of the Transformer and suggests the types of sequential relationships it is best suited for approximating. In particular, the results on approximation rates enable us to concretely analyze the differences between the Transformer and classical sequence modeling methods, such as recurrent neural networks.

Approximation Rate of the Transformer Architecture for Sequence Modeling

TL;DR

This work develops Jackson-type approximation rate results for a simplified, single-layer Transformer with one head, linking the model’s approximation power to a low-rank temporal coupling in target functions. By decomposing targets into pairwise temporal interactions and element-wise components via a POD-based temporal coupling, and by defining suitable complexity measures (, , ), the authors derive explicit error bounds showing how the rate improves as the attention dimension and feed-forward budget grow. The representation theorem confirms that any target can be expressed in the proposed target form, and numerical demonstrations with synthetic targets and ViT on CIFAR-10 support the theoretical insights, including practical low-rank patterns in real data. A comparative analysis with RNNs reveals that Transformers can be robust to temporal ordering changes but can be sensitive to temporal mixing, implying complementary strengths between the architectures. Collectively, the results illuminate when Transformers excel in sequence modeling and guide future work toward multi-heads, deeper architectures, and input preprocessing to mitigate temporal mixing.

Abstract

The Transformer architecture is widely applied in sequence modeling applications, yet the theoretical understanding of its working principles remains limited. In this work, we investigate the approximation rate for single-layer Transformers with one head. We consider a class of non-linear relationships and identify a novel notion of complexity measures to establish an explicit Jackson-type approximation rate estimate for the Transformer. This rate reveals the structural properties of the Transformer and suggests the types of sequential relationships it is best suited for approximating. In particular, the results on approximation rates enable us to concretely analyze the differences between the Transformer and classical sequence modeling methods, such as recurrent neural networks.
Paper Structure (44 sections, 10 theorems, 81 equations, 5 figures, 3 tables)

This paper contains 44 sections, 10 theorems, 81 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Sequence modeling as an approximation problem
  4. Motivation of Jackson-type Approximation Rates
  5. Formulation of Sequence Modeling as Approximation Problems
  6. The Transformer Hypothesis Space.
  7. Approximation results
  8. Permutation Equivariance and Position Encodings
  9. Jackson-type approximation rate
  10. Representation of the target space ${\mathcal{C}}$
  11. Temporal Coupling Component
  12. Element-wise Component
  13. Approximation Rates
  14. Numerical Demonstrations
  15. Practical Example
  16. ...and 29 more sections

Key Result

Theorem 4.1

Consider $d$-dimensional, length $\tau$ input space ${\mathcal{X}}^{(E)}$ with position encoding added. Then, for any $\bm H\in C({\mathcal{X}}^{(E)}, {\mathcal{Y}})$, there exists continuous functions $F\in C([0,1]^n, {\mathbb R})$, $f\in C({\mathcal{I}}, [0,1]^n)$ and $\rho \in C({\mathcal{I}} \ti where $n=2\tau d + 1$ and $\sigma$ is the softmax function. The proof is presented in appen: densit

Figures (5)

  • Figure 1: (a) is the estimated singular value of the attention matrix over a set of inputs for $m_h=64$. The violin plot shows the distribution of each singular value. (b) plots the estimated singular values for models with different $m_h$. (c) plots the training error against $m_h$.
  • Figure 2: For Figures (a) and (b) we examine a particular instance of the input $\bm x$. Figure (a) plots the attention matrix $A$, while Figure (b) illustrates the learned relationships, with green points and red points representing points from set $U$ and $V$, respectively. Figure (c) is the scatter plot of $F\circ f(\bm x)$ for randomly generated inputs $\bm x$.
  • Figure 3: Figure (a) plots the attention matrix $A$, while Figure (b) is the illustration of the learned relationship. In this instance, there are two sequences, $\bm x_1$ and $\bm x_2$, each connected to their respective predictions. The color of the connecting lines represents the corresponding values in $A$. Figure (c) presents the contour plot of $F\circ f(\bm x)$, generated for a set of random inputs $\bm x$.
  • Figure 4: Figure (a) plots the attention matrix $A$, while Figure (b) illustrates the learned relationship. The points in different colors refer to the input and output, respectively. They are connected based on the value of $A$. Figure (c) presents the scatter plot of $F\circ f(\bm x)$, generated for a set of random inputs $\bm x$.
  • Figure 5: We consider two class of $\rho$ with different singular value decay rate $\alpha$ as indicated above. Here, $r$ denotes the rank of the target. For each (a) and (b), we consider three targets with $\rho$ having different ranks, where $r = 2,6,\infty$. The figure plots the training error against $m_h$. Each colored line corresponds to a target with rank $r$ as indicated in the legend. The grey dotted line plots $m_h^{-(2\alpha-1)}$.

Theorems & Definitions (16)

  • Theorem 4.1: Representation of the target space
  • Theorem 4.2: Jackson-type approximation rates for the Transformer
  • Proposition 6.1
  • Proposition 6.2
  • Proposition 6.3
  • Theorem A.1: Kolmogorov Representation Theorem
  • Proposition A.2
  • Theorem A.3
  • proof
  • Lemma A.4
  • ...and 6 more