Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A unified framework for establishing the universal approximation of transformer-type architectures

Jingpu Cheng, Ting Lin, Zuowei Shen, Qianxiao Li

TL;DR

The paper presents a unified, verifiable framework for the universal approximation property (UAP) of transformer-type architectures, unifying analysis across kernel-based, sparse, and other attention mechanisms. Central to the theory are a nonlinear, affine-invariant feedforward family and a token-mixing mechanism that can distinguish tokens under a permutation group G; when these conditions hold, the transformer family achieves $G$-UAP in $L^p(K)$ for compact sets. A key technical advance is reducing the token-distinguishability check to a two-sample test under an analytic parameterization, enabling broad applicability to diverse attention forms. The framework informs principled design of symmetry-aware and novel attention mechanisms (including those with bias terms and particular $D_n$ or $C_n$ symmetries) with guaranteed UAP, providing a non-constructive yet verifiable foundation for expressive transformer architectures.

Abstract

We investigate the universal approximation property (UAP) of transformer-type architectures, providing a unified theoretical framework that extends prior results on residual networks to models incorporating attention mechanisms. Our work identifies token distinguishability as a fundamental requirement for UAP and introduces a general sufficient condition that applies to a broad class of architectures. Leveraging an analyticity assumption on the attention layer, we can significantly simplify the verification of this condition, providing a non-constructive approach in establishing UAP for such architectures. We demonstrate the applicability of our framework by proving UAP for transformers with various attention mechanisms, including kernel-based and sparse attention mechanisms. The corollaries of our results either generalize prior works or establish UAP for architectures not previously covered. Furthermore, our framework offers a principled foundation for designing novel transformer architectures with inherent UAP guarantees, including those with specific functional symmetries. We propose examples to illustrate these insights.

A unified framework for establishing the universal approximation of transformer-type architectures

TL;DR

The paper presents a unified, verifiable framework for the universal approximation property (UAP) of transformer-type architectures, unifying analysis across kernel-based, sparse, and other attention mechanisms. Central to the theory are a nonlinear, affine-invariant feedforward family and a token-mixing mechanism that can distinguish tokens under a permutation group G; when these conditions hold, the transformer family achieves -UAP in for compact sets. A key technical advance is reducing the token-distinguishability check to a two-sample test under an analytic parameterization, enabling broad applicability to diverse attention forms. The framework informs principled design of symmetry-aware and novel attention mechanisms (including those with bias terms and particular or symmetries) with guaranteed UAP, providing a non-constructive yet verifiable foundation for expressive transformer architectures.

Abstract

We investigate the universal approximation property (UAP) of transformer-type architectures, providing a unified theoretical framework that extends prior results on residual networks to models incorporating attention mechanisms. Our work identifies token distinguishability as a fundamental requirement for UAP and introduces a general sufficient condition that applies to a broad class of architectures. Leveraging an analyticity assumption on the attention layer, we can significantly simplify the verification of this condition, providing a non-constructive approach in establishing UAP for such architectures. We demonstrate the applicability of our framework by proving UAP for transformers with various attention mechanisms, including kernel-based and sparse attention mechanisms. The corollaries of our results either generalize prior works or establish UAP for architectures not previously covered. Furthermore, our framework offers a principled foundation for designing novel transformer architectures with inherent UAP guarantees, including those with specific functional symmetries. We propose examples to illustrate these insights.

Paper Structure

This paper contains 31 sections, 13 theorems, 83 equations.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Transformer architecture
  4. Universal approximation under permutation equivariance
  5. Main results
  6. Applications to practical architectures
  7. Kernel-based attention
  8. Sparse attention
  9. Other attention mechanisms
  10. New attention mechanisms from the approximation analysis
  11. New attention mechanism with bias term
  12. Transformer with UAP under specific symmetry
  13. Architecture with $D_n$-symmetry
  14. Architecture with $C_n$-symmetry:
  15. Conclusion
  16. ...and 16 more sections

Key Result

Theorem 1

Suppose that $\mathcal{H}$ is nonlinear and affine-invariant def:nonlinearity, and $\mathcal{G}$ satisfies the token distinguishability condition def:token_distinguishability. Then, the family of transformers $\mathcal{T}_{\mathcal{G}, \mathcal{H}}$ satisfies the $G$-UAP def:G-UAP.

Theorems & Definitions (27)

  • Definition 1: $G$-UAP
  • Definition 2: Nonlinearity and affine-invariance for $\mathcal{H}$
  • Definition 3: Token distinguishability for $\mathcal{G}$
  • Theorem 1
  • Theorem 2
  • Corollary 1
  • Definition 4
  • Corollary 2
  • Corollary 3
  • Corollary 4
  • ...and 17 more