Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unveiling and Controlling Anomalous Attention Distribution in Transformers

Ruiqing Yan, Xingbo Du, Haoyu Deng, Linghan Zheng, Qiuzhuang Sun, Jifang Hu, Yuhang Shao, Penghao Jiang, Jinrong Jiang, Lian Zhao

TL;DR

This work investigates the anomalous emphasis on the first element in Transformer attention using a waiver perspective, which posits that some sequence positions reduce their internal contributions in Softmax to absorb excess attention. It identifies two selection strategies—positional-encoding-based and feature-distribution-within-elements-based—and demonstrates how to controllably designate waiver elements via a structured mask or by adjusting positional-encoding distributions. Experiments on Llama3-8B (causal attention) and Bert-Large (global attention) validate that the proposed controls induce targeted attention patterns and can stabilize inference in long sequences. The findings offer a coherent explanation for attention sinks and point to practical gains in KV cache compression and infinite extrapolation, with implications for efficient and robust Transformer deployments.

Abstract

With the advent of large models based on the Transformer architecture, researchers have observed an anomalous phenomenon in the Attention mechanism--there is a very high attention on the first element, which is prevalent across Transformer-based models. It is crucial to understand it for the development of techniques focusing on attention distribution, such as Key-Value (KV) Cache compression and infinite extrapolation; however, the latent cause leaves to be unknown. In this paper, we analyze such a phenomenon from the perspective of waiver phenomenon, which involves reducing the internal values of certain elements in the sequence, allowing them to absorb excess attention without affecting their contribution to information. In specific models, due to differences in positional encoding and attention patterns, we have found that the selection of waiver elements by the model can be categorized into two methods: positional-encoding-based and feature-distribution-within-elements-based.

Unveiling and Controlling Anomalous Attention Distribution in Transformers

TL;DR

This work investigates the anomalous emphasis on the first element in Transformer attention using a waiver perspective, which posits that some sequence positions reduce their internal contributions in Softmax to absorb excess attention. It identifies two selection strategies—positional-encoding-based and feature-distribution-within-elements-based—and demonstrates how to controllably designate waiver elements via a structured mask or by adjusting positional-encoding distributions. Experiments on Llama3-8B (causal attention) and Bert-Large (global attention) validate that the proposed controls induce targeted attention patterns and can stabilize inference in long sequences. The findings offer a coherent explanation for attention sinks and point to practical gains in KV cache compression and infinite extrapolation, with implications for efficient and robust Transformer deployments.

Abstract

With the advent of large models based on the Transformer architecture, researchers have observed an anomalous phenomenon in the Attention mechanism--there is a very high attention on the first element, which is prevalent across Transformer-based models. It is crucial to understand it for the development of techniques focusing on attention distribution, such as Key-Value (KV) Cache compression and infinite extrapolation; however, the latent cause leaves to be unknown. In this paper, we analyze such a phenomenon from the perspective of waiver phenomenon, which involves reducing the internal values of certain elements in the sequence, allowing them to absorb excess attention without affecting their contribution to information. In specific models, due to differences in positional encoding and attention patterns, we have found that the selection of waiver elements by the model can be categorized into two methods: positional-encoding-based and feature-distribution-within-elements-based.
Paper Structure (19 sections, 11 equations, 4 figures)

This paper contains 19 sections, 11 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transformers
  4. Attention Sinks
  5. Analysis from a Waiver Perspective
  6. Waiver and its Latent Cause
  7. Properties of Waiver Elements
  8. How Waiver Elements Work in the Model
  9. Models Using Causal Attention
  10. Models Using Global Attention and Learnable Positional Encoding
  11. Experiment
  12. Protocols
  13. Experimental Design
  14. Adjusting the Structured Mask Matrix
  15. Adjusting the Feature Distribution within Positional Encodings
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The values of the elements of the non-mixture distribution in the FFN output of the second layer of the Llama3-8B model.
  • Figure 2: The L2 norms of the embedding vectors in the learnable positional encoding of the Bert-Large model.
  • Figure 3: In each subfigure, points of the same color represent elements within a sequence. (a) Special start tokens, the element at position index 255 is not changed; (b) Special start tokens, the element at position index 255 is changed; (c) Random start tokens, the element at position index 255 is not changed; (d) Random start tokens, the element at position index 255 is changed.
  • Figure 4: In each subfigure, points of the same color represent elements within a sequence. (a) Special start and end tokens, the element at position index 255 is not changed; (b) Special start and end tokens, the element at position index 255 is changed by first position embedding; (c) Random start and end tokens, the element at position index 255 is not changed; (d) Random start and end tokens, the element at position index 255 is changed by first position embedding; (e) Special start and end tokens, the element at position index 255 is not changed; (f) Special start and end tokens, the element at position index 255 is changed by last position embedding; (g) Random start and end tokens, the element at position index 255 is not changed; (h) Random start and end tokens, the element at position index 255 is changed by last position embedding.