Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformers are Expressive, But Are They Expressive Enough for Regression?

Swaroop Nath, Harshad Khadilkar, Pushpak Bhattacharyya

TL;DR

The paper investigates whether Transformers are universal function approximators, challenging prevailing claims by showing they struggle to approximate smooth functions and instead rely on piecewise-constant representations with small δ. It combines a theoretical bound—demonstrating that achieving good approximation forces a resolution factor δ that drives the layer count to $O(m(1/\delta)^{dm})$—with extensive synthetic experiments on both regression of smooth functions and classification of piecewise-constant targets. The experimental results with the full Transformer corroborate the theory for the encoder, showing limited ability to approximate smooth functions, while achieving relatively better performance on coarse piecewise-constant tasks. The work highlights fundamental expressivity limits of standard Transformer architectures and motivates architectural changes to attain universal function approximation in practice.

Abstract

Transformers have become pivotal in Natural Language Processing, demonstrating remarkable success in applications like Machine Translation and Summarization. Given their widespread adoption, several works have attempted to analyze the expressivity of Transformers. Expressivity of a neural network is the class of functions it can approximate. A neural network is fully expressive if it can act as a universal function approximator. We attempt to analyze the same for Transformers. Contrary to existing claims, our findings reveal that Transformers struggle to reliably approximate smooth functions, relying on piecewise constant approximations with sizable intervals. The central question emerges as: ''Are Transformers truly Universal Function Approximators?'' To address this, we conduct a thorough investigation, providing theoretical insights and supporting evidence through experiments. Theoretically, we prove that Transformer Encoders cannot approximate smooth functions. Experimentally, we complement our theory and show that the full Transformer architecture cannot approximate smooth functions. By shedding light on these challenges, we advocate a refined understanding of Transformers' capabilities. Code Link: https://github.com/swaroop-nath/transformer-expressivity.

Transformers are Expressive, But Are They Expressive Enough for Regression?

TL;DR

The paper investigates whether Transformers are universal function approximators, challenging prevailing claims by showing they struggle to approximate smooth functions and instead rely on piecewise-constant representations with small δ. It combines a theoretical bound—demonstrating that achieving good approximation forces a resolution factor δ that drives the layer count to —with extensive synthetic experiments on both regression of smooth functions and classification of piecewise-constant targets. The experimental results with the full Transformer corroborate the theory for the encoder, showing limited ability to approximate smooth functions, while achieving relatively better performance on coarse piecewise-constant tasks. The work highlights fundamental expressivity limits of standard Transformer architectures and motivates architectural changes to attain universal function approximation in practice.

Abstract

Transformers have become pivotal in Natural Language Processing, demonstrating remarkable success in applications like Machine Translation and Summarization. Given their widespread adoption, several works have attempted to analyze the expressivity of Transformers. Expressivity of a neural network is the class of functions it can approximate. A neural network is fully expressive if it can act as a universal function approximator. We attempt to analyze the same for Transformers. Contrary to existing claims, our findings reveal that Transformers struggle to reliably approximate smooth functions, relying on piecewise constant approximations with sizable intervals. The central question emerges as: ''Are Transformers truly Universal Function Approximators?'' To address this, we conduct a thorough investigation, providing theoretical insights and supporting evidence through experiments. Theoretically, we prove that Transformer Encoders cannot approximate smooth functions. Experimentally, we complement our theory and show that the full Transformer architecture cannot approximate smooth functions. By shedding light on these challenges, we advocate a refined understanding of Transformers' capabilities. Code Link: https://github.com/swaroop-nath/transformer-expressivity.
Paper Structure (19 sections, 1 theorem, 21 equations, 10 figures, 3 tables)

This paper contains 19 sections, 1 theorem, 21 equations, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Notations, Definitions and Preliminaries
  4. Transformer Architecture
  5. Necessary Definitions
  6. Transformers as Universal Function Approximators
  7. Expressivity of Transformers in the Continuous Space
  8. Proof Sketch for Theorem \ref{['thm:resolution-factor-bound']}
  9. Experiments
  10. Evaluation Measures
  11. Synthetic Datasets & Design Considerations
  12. Implementation, Results & Analysis
  13. Summary, Conclusion and Future Work
  14. Limitations
  15. Assumptions of our Proofs
  16. ...and 4 more sections

Key Result

Theorem 4.1

Let $\overline{f}$ ($\in \mathcal{\overline{F}}$) be an approximation to a smooth function f ($\in \mathcal{F}$), with goodness of approximation defined by $d_p$($\overline{f}$, $f$) $\leq$$\epsilon$ ($\epsilon > 0$). Then an upper bound on $\delta$ can be expressed as: where $\mathcal{X}^0$ is a covering over the compact support ($\mathcal{S}$) for the function $f$, $p$ is the norm in $d_p$, $d$

Figures (10)

  • Figure 1: Effect of changing the size of resolution factor. In (a) we have a smaller resolution factor, leading to a smaller error in approximation than (b).
  • Figure 2: (a) Behaviour of $\overline{f}(x)$ in the $\delta/2$ neighborhood of $x^0$. In this neighborhood, we can approximate $f(x)$ by $f(x^0)$, hence $\overline{f}(x) = f(x^0)$. (b) The shaded area in $\delta/2$ neighborhood of $x^0$ for $f(x)$ can be approximated by two triangles.
  • Figure 3: Trend of failure-rate with respect to the number of layers ($L$) and number of attention heads ($h$) of the Transformer. We perform each experiment (training-validation-testing pipeline) $5$ times. The line in the graph corresponds to the mean across runs, and the bands around the line indicate the standard deviation. For Expt-I, we keep $r=d=32$, and for Expt-II, we keep $r=d=128$. These configurations were found to be best performing from Figure \ref{['fig:trend-dims']}.
  • Figure 4: Trend of failure-rate with respect to embedding dimension ($d$) and the dimension of token-wise feed-forward network ($r$) of the Transformer. We perform each experiment (training-validation-testing pipeline) $5$ times. The line in the graph corresponds to the mean across runs, and the bands around the line indicate the standard deviation.
  • Figure 5: Trend of failure-rate with respect to the number of inputs and the number of outputs. We perform each experiment (training-validation-testing pipeline) $5$ times. The colored bars correspond to the mean across runs, while the black lines indicate the standard deviation. For Expt-I, we keep $r=d=32$, and for Expt-II, we keep $r=d=128$. These configurations were found to be best performing from Figure \ref{['fig:trend-dims']}.
  • ...and 5 more figures

Theorems & Definitions (6)

  • Definition 3.1: Resolution Factor
  • Definition 3.2: Adequacy of Approximation
  • Definition 3.3: Function with Compact Support
  • Theorem 4.1
  • proof
  • proof