Garbage Attention in Large Language Models: BOS Sink Heads and Sink-aware Pruning

TL;DR

The paper identifies a BOS sink attention mechanism as a major source of redundancy in deeper layers of LLMs and links this to the observed layer-wise pruning sensitivity. It introduces BOS sink scores and related metrics, and demonstrates that heads with high BOS sink scores are largely functionally redundant, particularly in deeper layers, through head ablations and attention analysis. A BOS-based structured pruning method is proposed, ranking heads and layers by BOS sink scores and achieving strong performance retention on Gemma-3-4B, Llama-3.1-8B, and Qwen3-4B even at aggressive pruning levels, outperforming magnitude- and activation-based baselines. The results suggest that structural properties of attention offer a robust, hardware-friendly basis for model compression, with stable behavior across sequence lengths and a potential for broader applicability to other transformer-based architectures.

Abstract

Large Language Models (LLMs) are known to contain significant redundancy, yet a systematic explanation for why certain components, particularly in higher layers, are more redundant has remained elusive. In this work, we identify the BOS sink phenomenon as a key mechanism driving this layer-wise sensitivity. We show that attention heads with high BOS sink scores are strongly associated with functional redundancy: such heads, especially in deeper layers, contribute little to predictive performance and effectively serve as \emph{dumping grounds} for superfluous attention weights. This provides a concrete functional explanation for the structural redundancy reported in prior studies. Leveraging this insight, we introduce a simple pruning strategy that removes high-BOS sink heads. Experiments on Gemma-3, Llama-3.1, and Qwen3 demonstrate that this approach identifies redundant transformer components more reliably than weight- or activation-based criteria, while preserving performance close to dense baselines even under aggressive pruning. Moreover, we find that the behavior of sink heads remains stable across different sequence lengths. Overall, our results suggest that structural properties of attention offer a more intuitive and robust basis for model compression than magnitude-based methods.

Figures (10)

• Figure 1: Attention weights of a representative <BOS> sink head (L26, H0) in Gemma-3-4B. The heatmap reveals a stark attention sink pattern where query tokens disproportionately attend to the <BOS> token during an MMLU task.
• Figure 2: Impact of single-head ablation on MMLU accuracy relative to <BOS> Sink Scores across Gemma-3-4B, Llama-3.1-8B and Qwen3-4B. Each dot represents an individual attention head, while the horizontal dashed line indicates the baseline MMLU accuracy for each respective model. A consistent pattern emerges across all three architectures: heads with high <BOS> Sink Scores exhibit negligible impact on model performance when ablated.
• Figure 3: Correlation between <BOS> sink scores and head ablation impact on MMLU accuracy. Colored and black lines represent layer-average sink scores (right axis) and mean MMLU accuracy under single-head ablation (left axis), respectively. In deeper layers, high sink scores strongly correlate with minimal functional necessity, indicating head redundancy.
• Figure 4: Scaling of average <BOS> sink scores with sequence length $T$. Each point represents the mean <BOS> sink score across all attention heads for a specific input. Dashed, dash-dotted, and solid lines denote linear regressions for all heads, heads with $\mu \ge 0.6$, and $\mu \ge 0.8$, respectively. Despite the $1/T$ softmax scaling effect, specialized sink heads (higher $\mu$) exhibit markedly more gradual slopes, indicating greater robustness and more consistent <BOS> focus as context length increases.
• Figure 5: Relationship between the mean <BOS> sink score ($\mu$) and the Coefficient of Variation ($CV$). Each dot represents an individual attention head, color-coded by its layer index. A clear inverse correlation is observed: heads with higher $\mu$ exhibit lower $CV$ values. Notably, these consistent <BOS> sink heads are primarily situated in the higher layers of the model, highlighting that functional stability and structural redundancy are deeply intertwined with the model's depth.