Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression

Weiyi He, Yue Xing

TL;DR

This work develops a Rademacher-complexity framework to quantify how completely learnable positional encodings (PE) affect the generalization and robustness of a one-layer Transformer in in-context regression. It derives clean RC bounds showing the generalization gap grows with the PE-induced parameter space and demonstrates an adversarial extension via Adversarial Rademacher Complexity (ARC), where attacks magnify the PE-related gap. A surrogate-loss framework and a detailed analysis of attack-induced changes to the solution space reveal a Phi(eps,t,d) factor that governs adversarial effects and context-length dependence. Empirical simulations corroborate the theory, showing larger gaps under PE, with longer contexts mitigating but not eliminating vulnerability to adversarial prompts, and RoPE behaving similarly to the no-PE baseline.

Abstract

Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer's generalization and robustness remains unclear. In this work, we provide the first generalization analysis for a single-layer Transformer under in-context regression that explicitly accounts for a completely trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.

Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression

TL;DR

This work develops a Rademacher-complexity framework to quantify how completely learnable positional encodings (PE) affect the generalization and robustness of a one-layer Transformer in in-context regression. It derives clean RC bounds showing the generalization gap grows with the PE-induced parameter space and demonstrates an adversarial extension via Adversarial Rademacher Complexity (ARC), where attacks magnify the PE-related gap. A surrogate-loss framework and a detailed analysis of attack-induced changes to the solution space reveal a Phi(eps,t,d) factor that governs adversarial effects and context-length dependence. Empirical simulations corroborate the theory, showing larger gaps under PE, with longer contexts mitigating but not eliminating vulnerability to adversarial prompts, and RoPE behaving similarly to the no-PE baseline.

Abstract

Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer's generalization and robustness remains unclear. In this work, we provide the first generalization analysis for a single-layer Transformer under in-context regression that explicitly accounts for a completely trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.

Paper Structure

This paper contains 46 sections, 21 theorems, 120 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. ICL Task and Transformer Architecture
  5. Training and Adversarial Loss
  6. Generalization Framework
  7. Main Results
  8. RC for Transformer without PE
  9. RC for Transformer with PE
  10. ARC for Transformer
  11. The Surrogate Loss Framework.
  12. The Impact of Attack on Solution Space.
  13. Simulations
  14. Setup
  15. Standard Generalization Bounds
  16. ...and 31 more sections

Key Result

Proposition 3.1

Let $\mathcal{H}$ be a hypothesis class and $\ell$ be a loss function. If the magnitude of our loss function is bounded above by $c$, with probability at least $1-\delta$: where $\ell \circ \mathcal{H}$ is the class of functions formed by composing the loss with the hypotheses.

Figures (3)

  • Figure 1: Mean Generalization Gap vs. Context Length ($t$) for models with and without Positional Encodings (PE).
  • Figure 2: Mean Attacked Generalization Gap vs. Context Length ($t$) under a fixed PGD attack ($\varepsilon=0.02$).
  • Figure 3: Mean Attacked Generalization Gap vs. Context Length ($t$) for different attack strengths ($\varepsilon$).

Theorems & Definitions (43)

  • Proposition 3.1: Generalization Bound
  • Theorem 4.1: RC for Transformer without PE
  • Lemma 4.1: Generalization Bound via Dudley's Integral
  • Definition 4.1: Effective Parameter Space
  • Lemma 4.2: Covering Number of the Parameter Space
  • Lemma 4.3
  • Lemma 4.4: Justification for $L_G$ and $r$
  • Theorem 4.2: RC with PE
  • Proposition 4.1: Surrogate Loss Upper Bound
  • Proposition 4.2: Complexity Bound via Surrogate
  • ...and 33 more