Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demystifying the Slash Pattern in Attention: The Role of RoPE

Yuan Cheng, Fengzhuo Zhang, Yunlong Hou, Cunxiao Du, Chao Du, Tianyu Pang, Aixin Sun, Zhuoran Yang

TL;DR

This work uncovers the mechanisms behind slash-dominant attention heads (SDHs) in RoPE-enabled Transformers. Through thorough empirical analysis of open-source LLMs, it shows SDHs are intrinsic to the model architecture and robust to out-of-distribution prompts, with pre-PE queries/keys displaying near rank-one structure and RoPE frequencies driving the cross-token variation. A Fourier-like decomposition of attention logits reveals that high- and medium-frequency RoPE components principally shape the slash pattern, while token embeddings lie on a cone, explaining the near-constancy of Q/K directions across tokens. Theoretical results for a shallow RoPE Transformer trained via gradient descent establish two-stage learning dynamics that produce an SDH and prove its generalization to OOD inputs, with extensions to large offsets. Collectively, the paper offers actionable insights into the role of RoPE in information propagation, suggesting avenues for parameter-efficient Q/K representations and length-generalization strategies in long-context generation.

Abstract

Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the $Δ$-th sub-diagonal for some offset $Δ$. These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.

Demystifying the Slash Pattern in Attention: The Role of RoPE

TL;DR

This work uncovers the mechanisms behind slash-dominant attention heads (SDHs) in RoPE-enabled Transformers. Through thorough empirical analysis of open-source LLMs, it shows SDHs are intrinsic to the model architecture and robust to out-of-distribution prompts, with pre-PE queries/keys displaying near rank-one structure and RoPE frequencies driving the cross-token variation. A Fourier-like decomposition of attention logits reveals that high- and medium-frequency RoPE components principally shape the slash pattern, while token embeddings lie on a cone, explaining the near-constancy of Q/K directions across tokens. Theoretical results for a shallow RoPE Transformer trained via gradient descent establish two-stage learning dynamics that produce an SDH and prove its generalization to OOD inputs, with extensions to large offsets. Collectively, the paper offers actionable insights into the role of RoPE in information propagation, suggesting avenues for parameter-efficient Q/K representations and length-generalization strategies in long-context generation.

Abstract

Large Language Models (LLMs) often exhibit slash attention patterns, where attention scores concentrate along the -th sub-diagonal for some offset . These patterns play a key role in passing information across tokens. But why do they emerge? In this paper, we demystify the emergence of these Slash-Dominant Heads (SDHs) from both empirical and theoretical perspectives. First, by analyzing open-source LLMs, we find that SDHs are intrinsic to models and generalize to out-of-distribution prompts. To explain the intrinsic emergence, we analyze the queries, keys, and Rotary Position Embedding (RoPE), which jointly determine attention scores. Our empirical analysis reveals two characteristic conditions of SDHs: (1) Queries and keys are almost rank-one, and (2) RoPE is dominated by medium- and high-frequency components. Under these conditions, queries and keys are nearly identical across tokens, and interactions between medium- and high-frequency components of RoPE give rise to SDHs. Beyond empirical evidence, we theoretically show that these conditions are sufficient to ensure the emergence of SDHs by formalizing them as our modeling assumptions. Particularly, we analyze the training dynamics of a shallow Transformer equipped with RoPE under these conditions, and prove that models trained via gradient descent exhibit SDHs. The SDHs generalize to out-of-distribution prompts.
Paper Structure (79 sections, 32 theorems, 146 equations, 39 figures, 13 tables, 1 algorithm)

This paper contains 79 sections, 32 theorems, 146 equations, 39 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Preliminaries
  4. Empirical Study of Slash-Dominance of Attention
  5. OOD Generalization of sdh
  6. Approximate Low-Rankness of pre-pe Queries and Keys
  7. How to Get Approximately Rank-One Queries and Keys?
  8. Collaboration of Frequencies in rope Determines Slash Pattern
  9. Theoretical Study of Shallow Transformers
  10. Data Model
  11. Slash-Dominance Frequency Condition
  12. Token Embedding.
  13. Network Architecture
  14. Training Settings
  15. Loss Function.
  16. ...and 64 more sections

Key Result

Theorem 5.3

Suppose Assumption Assp 1: Frequency seq holds. Consider $N \gg \mathrm{poly}(K) \gg \mathrm{polylog}(N)$, $N \gg d$, and $p_k=1/K$ for any $k \in [K]$. We set $\epsilon_{1}=O({N^{-\frac{1}{2}}})$ and $\epsilon_{2}=O({N^{-\frac{1}{4}}})\bigcap \Omega(\epsilon_{1})$ as the attention concentration err the attention head of the first layer is trained to be $1-\epsilon_{1}$ slash-dominant at $1$. Form

Figures (39)

  • Figure 1: Illustration of the emergence of sdh. Attention scores are determined by pre-PE queries, keys, and rope (left bottom). Because token embeddings lie approximately on a cone, queries/keys are almost rank-one, and nearly identical across tokens (left top), so rope primarily governs variation of attention scores across tokens. Then rope's high- and medium-frequency components interact constructively at specific lags, producing the attention score peaks at offset $\Delta$ (right top). As a result, sdh emerge and are ood generalizable (right bottom).
  • Figure 2: Mind map of our empirical studies.
  • Figure 3: Average of attention score matrices in Qwen2.5-7B-Instruct with prompts from LongBench. We denote the $a$-th head at $b$-th layer as $\mathrm{L} b \mathrm{H} a$ in this paper. In panels (a)–(c), attention concentrates on the sub-diagonals with small offsets $0,1$ and $2$, respectively. In panels (d)–(f), it also concentrates on sub-diagonals with large offsets exceeding $500$.
  • Figure 4: The forwarding head forwards semantic information from the prefix to the current token. In the feature-matching head, the target question token ("is") matches tokens whose prefixes exhibit similar semantics to the target, enabling the model to generate the correct continuation "PE".
  • Figure 5: Illustration of Slash-Dominance at $\Delta$.
  • ...and 34 more figures

Theorems & Definitions (34)

  • Definition 4.1: $(\kappa, \Delta)$-Slash-Dominance
  • Definition 5.2: Two-layer Disentangled Transformer
  • Theorem 5.3: Training Dynamics
  • Lemma F.1: Layer 1 Gradient
  • Lemma F.2: Track $A_{l,r}(t)$ Update
  • Lemma F.3: Characterization of $a_{i,j}$
  • Lemma F.4
  • Corollary F.5: Bounds of $I(i)$
  • Lemma F.6: Concentration of $S^{(2)}$,$\mathcal{S}^{(2)}$
  • Corollary F.7
  • ...and 24 more