Identifying and Evaluating Inactive Heads in Pretrained LLMs

TL;DR

This work investigates the prevalence of inactive attention heads in pretrained LLMs beyond traditional attention-weight metrics. By proposing a taxonomy of 13 score functions spanning attention patterns, value vectors, and head outputs, and applying dynamic thresholding plus model interventions on MMLU across 14 models, the authors demonstrate that a substantial fraction of heads (avg >12%) can be ablated with minimal accuracy loss. They show that head-output–based signals, especially Avg Head Output Norm with layer normalization (AHON-LN), robustly identify inactive heads across model families and scales, while first-token attention sinks alone underestimate inactivity. The study also reveals that finetuning induces little change in score distributions and that scale effects emerge primarily at very large sizes, suggesting a degree of invariance in attention head behavior to common training modifications. These findings have implications for efficient inference and model compression by targeting dormant heads identified via head-output metrics rather than attention weights.

Abstract

Attention is foundational to large language models (LLMs), enabling different heads to have diverse focus on relevant input tokens. However, learned behaviors like attention sinks, where the first token receives the most attention despite limited semantic importance, suggest some heads may be inactive, and point to a significant source of computational redundancy. To analyze this phenomenon, we propose a taxonomy of 13 score functions that measure different ways a head can be inactive. Thresholding these scores allows us to analyze different sets of potentially inactive attention heads. We evaluate whether identified heads are inactive through model interventions, finding that more than 12% of attention heads are inactive on average, and can be ablated in specific contexts while maintaining MMLU accuracy to within 1% of the pretrained LLM. Across 3 model families, our score functions that measure the average norm of a head's output consistently identify inactive heads that would not have been found by score functions that rely solely on attention weights. We establish that relying on a score function that measures a first token attention sink would underestimate the prevalence of inactive heads, failing to identify more than 7% of inactive heads on average. We also show how measuring score distributions can provide insights into attention behavior. For instance, we find evidence that finetuning causes little to no change in attention behavior, and that even within the same model family, large model scales present markedly different attention behaviors.

Figures (11)

• Figure 1: The Inactive Head Taxonomy. Our simple scoring functions measure all three components of attention: attention weights, value vectors, and head output vectors. In each cell, we give an example of the type of attention head that would be identified by each scoring function once scores are thresholded. In attention weights, color represents magnitude. For vectors, color represents direction, and length of the blue bars represents magnitude. For example, the Avg Weight of First Token score gu2025whenguo2024activedormantattentionheadsmechanistically is calculated by computing average weight to the first token, and heads that exhibit high attention to the first token are identified if their their score exceeds a threshold. Value vectors and head outputs do not play a role, which we illustrate with a fade. We also consider normalizing scores by the average score of other heads in the layer. Including normalized versions of these scoring functions, there are $13$ in total.
• Figure 2: Score functions identify different sets of heads. Identifying different sets of heads ensures we capture a broad range of head characteristics, rather than focusing solely on the attention sink pattern of Avg Weight of First Token gu2025when. Using $100$ FineWeb-Edu training samples, we measure IoU of classifications between each score function on Llama-3.1-8B. We also measure Precision of one scoring function's classifications relative to another's, using the column scoring function as ground truth. Column score functions are abbreviated using the first letter of each word, and are in the same order as row score functions. For each score function, we dynamically choose thresholds such that $\sim10\%$ of heads are identified as potentially inactive. Even head scores normalized by scores of other heads in the layer, denoted by "(LN)", do not show significant IoU or Precision with their unnormalized counterpart.
• Figure 3: Measuring Avg Head Output Norm is best at identifying inactive heads for most models. For every model, we rank the $13$ scoring functions by normalized AUC, which captures their ability to select inactive heads. Avg Head Output Norm (LN) ranks $1^{\text{st}}$ for 8 out of 14 models, and ranks in top-$3$ for 13 out of 14 models. Top scoring functions are consistent across model families.
• Figure 4: Inactive heads can be identified and zeroed with minor performance degradation. For each model, we plot the top-3 scoring functions. Average Weight of First Token is not in the top-3 for OLMo-2-1124-7B-Instruct, but we include it for comparison to prior work. The gray dotted line represents baseline accuracy of the pretrained model. The black line represents zeroing out heads uniformly at random. The Avg Head Output Norm (LN) scoring function is best at identifying the most heads, that when zeroed, maintain accuracy. Complete results for all models and score functions can be found in App. \ref{['fig:appendix-mmlu-accuracy-vs-percent-of-heads-zeroed']}.
• Figure 5: Head score distributions can be similar within a model family. Smaller Wasserstein distances indicate similar score distributions. Full results for all $13$ scoring functions can be found in App. \ref{['fig:appendix-wasserstein-dist-of-score-distributions']}.