On the Existence and Behaviour of Secondary Attention Sinks

TL;DR

This paper identifies secondary sinks, a class of attention sinks that arise in the middle layers of decoder-only transformers and persist for a limited number of layers, distinct from the BOS sink. It shows that these sinks are produced by specific middle-layer MLP modules whose outputs align with the BOS sink direction, and that the sink strength and lifetime scale with the $ extit{l_2}$-norm of these MLP outputs, forming discrete sink levels that vary with model size. The work demonstrates a compensatory relationship where the BOS sink weakens in middle layers as secondary sinks emerge, with larger models exhibiting more deterministic sink-level structure. These findings illuminate how attention distributions are shaped across depth and model scale, with potential implications for pretraining, post-training, and generation dynamics in large language models.

Abstract

Attention sinks are tokens, often the beginning-of-sequence (BOS) token, that receive disproportionately high attention despite limited semantic relevance. In this work, we identify a class of attention sinks, which we term secondary sinks, that differ fundamentally from the sinks studied in prior works, which we term primary sinks. While prior works have identified that tokens other than BOS can sometimes become sinks, they were found to exhibit properties analogous to the BOS token. Specifically, they emerge at the same layer, persist throughout the network and draw a large amount of attention mass. Whereas, we find the existence of secondary sinks that arise primarily in middle layers and can persist for a variable number of layers, and draw a smaller, but still significant, amount of attention mass. Through extensive experiments across 11 model families, we analyze where these secondary sinks appear, their properties, how they are formed, and their impact on the attention mechanism. Specifically, we show that: (1) these sinks are formed by specific middle-layer MLP modules; these MLPs map token representations to vectors that align with the direction of the primary sink of that layer. (2) The $\ell_2$-norm of these vectors determines the sink score of the secondary sink, and also the number of layers it lasts for, thereby leading to different impacts on the attention mechanisms accordingly. (3) The primary sink weakens in middle layers, coinciding with the emergence of secondary sinks. We observe that in larger-scale models, the location and lifetime of the sinks, together referred to as sink levels, appear in a more deterministic and frequent manner. Specifically, we identify three sink levels in QwQ-32B and six levels in Qwen3-14B.

Figures (5)

• Figure 1: (Bottom) In LLaMA2-7B-Chat, the 1st, 11th tokens act as primary sinks: they exhibit significantly larger $l_2$ norms in their hidden states, while having much smaller key and value norms compared to other tokens. (Top) DeepSeek-14B also shows primary sinks (tokens 0, 1, and 2). In addition, it exhibits other attention sinks that emerge in middle layers (e.g., tokens 1120, 2768).
• Figure 2: Sink levels ($l_\text{start}$, lifetime) at different model scales.
• Figure 3: Distribution of sinks across token index.
• Figure 4: (Left) Sink-score comparison between BOS and Secondary Sinks across whole model depth. (Mid)Secondary Sinks Sink-score grows log-linearly with $\ell_2$-norm $l_\text{start}$. (Right) Sink Lifetime growths with $\ell_2$-norm $l_\text{start}$.
• Figure 5: Swapping the activation of the secondary sink at hidden state ${\bm{h}}_t^l$, attention output ${\bm{o}}_t^l$, and MLP output ${\bm{f}}_t^l$ with that of an average uninformative token.