Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Rank-Dependent Generalisation Error Bounds for Transformers

Lan V. Truong

TL;DR

This paper introduces various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms, and applies these bounds to derive generalization errors for single layer transformers.

Abstract

In this paper, we introduce various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms. These bounds are contingent on the rank of each class of matrices. We then apply these bounds to derive generalization errors for single layer transformers. Our results improve upon several existing generalization bounds in the literature and are independent of input sequence length, highlighting the advantages of employing low-rank matrices in transformer design. More specifically, our achieved generalisation error bound decays as $O(1/\sqrt{n})$ where $n$ is the sample length, which improves existing results in research literature of the order $O((\log n)/(\sqrt{n}))$. It also decays as $O(\log r_w)$ where $r_w$ is the rank of the combination of query and and key matrices.

On Rank-Dependent Generalisation Error Bounds for Transformers

TL;DR

This paper introduces various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms, and applies these bounds to derive generalization errors for single layer transformers.

Abstract

In this paper, we introduce various covering number bounds for linear function classes, each subject to different constraints on input and matrix norms. These bounds are contingent on the rank of each class of matrices. We then apply these bounds to derive generalization errors for single layer transformers. Our results improve upon several existing generalization bounds in the literature and are independent of input sequence length, highlighting the advantages of employing low-rank matrices in transformer design. More specifically, our achieved generalisation error bound decays as where is the sample length, which improves existing results in research literature of the order . It also decays as where is the rank of the combination of query and and key matrices.

Paper Structure

This paper contains 19 sections, 16 theorems, 102 equations.

Table of Contents

  1. Introduction
  2. Background
  3. Matrix norms
  4. Generalisation Bounds
  5. Rademacher Complexity
  6. Covering Numbers
  7. Self-Attention and Transformers
  8. Rank-Dependent Covering Number Bounds for Linear Function Classes
  9. Rank-Dependent Generalisation Error Bounds for Single Layer Transformer
  10. Single Layer Single Head
  11. Single Layer Multiple Heads
  12. Comparison with existing research results
  13. Conclusion
  14. Proof of Theorem \ref{['thm:main2']}
  15. Proof of Theorem \ref{['thm:main3']}
  16. ...and 4 more sections

Key Result

Theorem 1

Let $r_w$ be a positive integer number and $\mathcal{V}_w$ be a sub-vector space of dimension $r_w$ of $\mathbb{R}^k$. Define $\mathcal{W}=\{W \in \mathbb{R}^{d\times k}: \hbox{col}(W) \subset \mathcal{V}_w, \|W \|_{2\to 2} \leq B_w \} \}$, $\mathcal{F}=\{x \to Wx: W \in \mathcal{W}\}$, and let o

Theorems & Definitions (19)

  • Theorem 1
  • Remark 2
  • Lemma 3
  • Theorem 4
  • Theorem 5
  • Corollary 6
  • Remark 7
  • Theorem 8
  • Corollary 9
  • Corollary 10
  • ...and 9 more