Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input

TL;DR

The theoretical results support the practical success of Transformers for high dimensional data and show that even if the smoothness changes depending on each input, Transformers can estimate the importance of features for each input and extract important features dynamically.

Abstract

Despite the great success of Transformer networks in various applications such as natural language processing and computer vision, their theoretical aspects are not well understood. In this paper, we study the approximation and estimation ability of Transformers as sequence-to-sequence functions with infinite dimensional inputs. Although inputs and outputs are both infinite dimensional, we show that when the target function has anisotropic smoothness, Transformers can avoid the curse of dimensionality due to their feature extraction ability and parameter sharing property. In addition, we show that even if the smoothness changes depending on each input, Transformers can estimate the importance of features for each input and extract important features dynamically. Then, we proved that Transformers achieve similar convergence rate as in the case of the fixed smoothness. Our theoretical results support the practical success of Transformers for high dimensional data.

Figures (4)

• Figure 1: The self-attention mechanism can attend by relative position. In this diagram, each token attend to the previous token and itself.
• Figure 2: For piecewise $\gamma$-smoothness, the position of important tokens depends on each input. We show important tokens in darker color. In the case of $X \in \Omega_j$, the most important token to $y_i$ is $x_{i-2}$ and in the case of $X \in \Omega_k$, $x_{i+2}$ is the most important. The self-attention mechanism can switch its attention (represented by black lines) depending on the importance of tokens.
• Figure 3: Two zebra images (left) and the corresponding images with 180 / 196 patches masked (right).
• Figure 4: The predicted probability of the correct class for the top left image in Fig. \ref{['fig:images']}. The predicted probability remains high even when most of the images are masked.

Theorems & Definitions (32)

• Definition 3.1: $\gamma$-smooth function class
• Definition 3.2: Mixed and anisotropic smoothness
• Definition 3.3: Piecewise $\gamma$-smooth function class
• Definition 3.4: importance function
• Theorem 4.2
• Remark 4.3
• Theorem 4.5
• Definition 5.1: Covering Number
• Theorem 5.2
• Theorem 5.3