Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

Qingyue Yang, Jie Wang, Xing Li, Yinqi Bai, Xialiang Tong, Huiling Zhen, Jianye Hao, Mingxuan Yuan, Bin Li

TL;DR

TAPPA introduces a unifying temporal framework to explain the emergence of diverse attention patterns in autoregressive LLMs by analyzing the temporal evolution of queries under RoPE. It defines a quantitative measure, $q$-similarity, to distinguish predictable versus unpredictable patterns and provides theoretical conditions for Re-access, Sequential, and Seasonal patterns arising from the joint effect of queries, keys, and RoPE. The framework leverages a channel-wise decomposition of attention and the RoPE relative-position identity $R_m^\top R_n = R_{m-n}$ to explain phenomena such as periodic diagonals and seasonal repeats. Empirically, TAPPA improves downstream efficiency for KV-cache compression and LLM pruning using a simple $q$-similarity based budget adjustment, demonstrating practical impact and enabling broader, theory-driven optimization of transformer inference.

Abstract

Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce \textbf{Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations} from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.

Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

TL;DR

TAPPA introduces a unifying temporal framework to explain the emergence of diverse attention patterns in autoregressive LLMs by analyzing the temporal evolution of queries under RoPE. It defines a quantitative measure, -similarity, to distinguish predictable versus unpredictable patterns and provides theoretical conditions for Re-access, Sequential, and Seasonal patterns arising from the joint effect of queries, keys, and RoPE. The framework leverages a channel-wise decomposition of attention and the RoPE relative-position identity to explain phenomena such as periodic diagonals and seasonal repeats. Empirically, TAPPA improves downstream efficiency for KV-cache compression and LLM pruning using a simple -similarity based budget adjustment, demonstrating practical impact and enabling broader, theory-driven optimization of transformer inference.

Abstract

Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink heads, and diagonal traces, yet these observations remain fragmented and lack a unifying explanation. To bridge this gap, we introduce \textbf{Temporal Attention Pattern Predictability Analysis (TAPPA), a unifying framework that explains diverse attention patterns by analyzing their underlying mathematical formulations} from a temporally continuous perspective. TAPPA both deepens the understanding of attention behavior and guides inference acceleration approaches. Specifically, TAPPA characterizes attention patterns as predictable patterns with clear regularities and unpredictable patterns that appear effectively random. Our analysis further reveals that this distinction can be explained by the degree of query self-similarity along the temporal dimension. Focusing on the predictable patterns, we further provide a detailed mathematical analysis of three representative cases through the joint effect of queries, keys, and Rotary Positional Embeddings (RoPE). We validate TAPPA by applying its insights to KV cache compression and LLM pruning tasks. Across these tasks, a simple metric motivated by TAPPA consistently improves performance over baseline methods. The code is available at https://github.com/MIRALab-USTC/LLM-TAPPA.
Paper Structure (56 sections, 5 theorems, 74 equations, 8 figures, 10 tables)

This paper contains 56 sections, 5 theorems, 74 equations, 8 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Attention Patterns
  4. The Role of Positional Encoding
  5. The Influence of Input Dynamics
  6. Background
  7. Attention Mechanism.
  8. RoPE.
  9. Decomposition View of Attention.
  10. RoPE Key Property.
  11. Attention Patterns.
  12. Why Predictable and Unpredictable Attention Patterns Exist
  13. Predictable Attention Patterns
  14. Re-access Pattern
  15. Connection to Attention Sink in the First Token.
  16. ...and 41 more sections

Key Result

Proposition 4.1

Let $q_t, q_{t+1} \in \mathbb{R}^d$ be consecutive queries, $K=[k_1, \dots, k_T]^\top$ the key matrix, and define the logits If $q_{t+1}-q_t$ has a large norm and is not orthogonal to all rotated keys $\{R_{t+1-j}k_j\}$, then the difference between the logit vectors $a_t$ and $a_{t+1}$ is necessarily large. In particular, there exist constants $c_1,c_2>0$ such that

Figures (8)

  • Figure 1: Overview of Temporal Attention Pattern Predictability Analysis (TAPPA) Framework. Left: Theoretical discoveries. Query self-similarity (q-similarity) affects the predictable and unpredictable patterns. Within the periodic sequential pattern, the slash interval is affected by the joint effect of queries, keys and RoPE. Right: Q-similarity is applied to downstream tasks and achieves consistant improvements.
  • Figure 2: TAPPA explains the formation of sparse attention patterns from a temporal continuity perspective. We first establish the fundamental Predictable and Unpredictable patterns in Sec. \ref{['sec:random_stable_patterns']}. We then detail the conditions that form the Re-access (Sec. \ref{['sec:reaccess']}), Sequential (Sec. \ref{['sec:sequential']}), Seasonal (Sec. \ref{['sec:seasonal']}), and Periodical Sequential (Sec. \ref{['sec:sequential_period']}) patterns in their dedicated sections.
  • Figure 3: Attention patterns at high and low Query similarity on the Llama and Qwen models. Stable patterns emerge under high similarity, whereas low similarity results in random patterns. There are random bright dots of critical keys in the second and fourth figures.
  • Figure 4: High self-similarity in Query (Q) and Key (K) matrices results in sequential attention patterns. An example from a Qwen-2.5 head (left) with high Q and K self-similarity (0.99 and 0.96) produces a strong diagonal pattern in the attention map (far right). This phenomenon is also observed in Llama-3.1 (center right).
  • Figure 5: An illustration of how RoPE configuration affects attention patterns. (a) and (b) show a sequential pattern with a dominant channel at $m=124$. In (c) and (d), we manually change the dominant channel to higher frequencies ($m=2$ and $m=5$), which causes periodic diagonals to emerge. In (e), we change the RoPE base from $c=1,000,000$ to $c=100,000$ with $m=5$.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Proposition 4.1
  • Theorem 5.1: Vertical Stability of Attention
  • Theorem 5.2: Sequential Patterns under High Self-similarity
  • Theorem 5.3: Periodic Sequential Pattern from a Dominant RoPE Channel
  • Theorem 5.4: Seasonal Attention Pattern from Periodic Keys and Dominant RoPE Channel
  • proof
  • proof
  • proof
  • proof