Systematic Outliers in Large Language Models

TL;DR

The paper investigates systematic outliers in Transformer-based large language models (LLMs), identifying three categories—activation, weight, and attention outliers—and demonstrates that their formation is driven by the softmax operation in self-attention. By analyzing their locations, distributions, and interdependencies, the authors reveal a cascading lifecycle where weight outliers induce activation outliers, which then propagate to attention computations and can affect non-outlier tokens. They propose explicit context-aware scaling as a mechanism to mitigate these outliers, providing both theoretical justification and empirical validation across GPT-2 variants and TinyLLaMA, with notable improvements in model compression robustness and convergence speed. The work further shows that softmax is a central culprit and that scaling factors can preserve performance while enabling more efficient deployment. Overall, the study offers a mechanistic understanding of LLM internals and practical strategies to enhance efficiency and stability of Transformer-based models.

Abstract

Outliers have been widely observed in Large Language Models (LLMs), significantly impacting model performance and posing challenges for model compression. Understanding the functionality and formation mechanisms of these outliers is critically important. Existing works, however, largely focus on reducing the impact of outliers from an algorithmic perspective, lacking an in-depth investigation into their causes and roles. In this work, we provide a detailed analysis of the formation process, underlying causes, and functions of outliers in LLMs. We define and categorize three types of outliers-activation outliers, weight outliers, and attention outliers-and analyze their distributions across different dimensions, uncovering inherent connections between their occurrences and their ultimate influence on the attention mechanism. Based on these observations, we hypothesize and explore the mechanisms by which these outliers arise and function, demonstrating through theoretical derivations and experiments that they emerge due to the self-attention mechanism's softmax operation. These outliers act as implicit context-aware scaling factors within the attention mechanism. As these outliers stem from systematic influences, we term them systematic outliers. Our study not only enhances the understanding of Transformer-based LLMs but also shows that structurally eliminating outliers can accelerate convergence and improve model compression. The code is avilable at https://github.com/an-yongqi/systematic-outliers.

Figures (24)

• Figure 1: Systematic outliers in LLaMA2-7B. Outliers are identified in four locations: activations (layer outputs $\mathbf{h}_{\ell}$ and down-projection inputs $\mathbf{x}_{\ell}^{\text{down}}$), weights (down-projection matrices $\mathbf{W}_{\ell}^{\text{down}}$), and attention (attention weights $\mathbf{A}^i_{\ell}$).
• Figure 2: Illustration of systematic outliers locations in LLMs.
• Figure 3: Distribution of activation outliers in $\mathbf{h}_{\ell}$ across layers, sequences, and feature dimensions.
• Figure 4: Distribution of activation outliers in $\mathbf{x}_{\ell}^{\text{down}}$ across layers, sequences, and feature dimensions.
• Figure 5: Distribution of weight outliers in $\mathbf{W}_{\ell}^{\text{down}}$ across layers, modules, and feature dimensions.