Rethinking the Outlier Distribution in Large Language Models: An In-depth Study

TL;DR

This work tackles outliers in large language models (LLMs) that hinder low-precision post-training quantization (PTQ) by conducting an operator-level empirical study of two outlier types: massive activations (MAs) and channel-wise outliers (CO). It identifies that MAs largely originate in early FFN stages and propagate via residual connections, and that COs arise from normalization and specific weight-channel interactions, notably outlier-triggering channels (OTCs) in $W_q$, which interact with activations to generate CO. The authors propose targeted smoothing strategies—removing TMAs at $y_6$ or reducing rescaling variation in gamma and selectively tuning OTCs in weights—to eliminate most outliers with minimal impact on accuracy, thereby facilitating more efficient, low-bit quantization. The findings provide practical, low-overhead guidance for improving PTQ performance in LLMs, with implications for edge deployment and specialized hardware support.

Abstract

Investigating outliers in large language models (LLMs) is crucial due to their significant impact on various aspects of LLM performance, including quantization and compression. Outliers often cause considerable quantization errors, leading to degraded model performance. Identifying and addressing these outliers can enhance the accuracy and efficiency of the quantization process, enabling smoother deployment on edge devices or specialized hardware. Recent studies have identified two common types of outliers in LLMs: massive activations and channel-wise outliers. While numerous quantization algorithms have been proposed to mitigate their effects and maintain satisfactory accuracy, few have thoroughly explored the root causes of these outliers in depth. In this paper, we conduct a comprehensive investigation into the formation mechanisms of these outliers and propose potential strategies to mitigate their occurrence. Ultimately, we introduce some efficient approaches to eliminate most massive activations and channel-wise outliers with minimal impact on accuracy.

Figures (9)

• Figure 1: Given a pretrained LLM, techniques are first applied to smooth out the outliers in its activations. The resulting model is then quantized, achieving superior accuracy.
• Figure 2: (a) Architecture of a LLM decoder block. (b), (c) and (d) show the architectures of self-attention block, standard FFN (conventional MLP), and gated FFN (GLU), respectively. The notations will be used throughout the rest sections.
• Figure 3: (a) One example of massive activation presented in the inputs $x_{1}$. (b) An example of outlier channel at position $x_{2}$ in the LLM. (c) The existence of outlier will lead to an output Y' different from the original output Y. (d) The normalization operations within LLM.
• Figure 4: Left: TMAs and FMAs within the input of LLaMA-13B across each layer. Right: after removing the MAs in residual connection, only TMA left.
• Figure 5: Left: TMAs and FMAs within the input of GPT-2 across each layer. Right: after removing the MAs in residual connection, only TMA left.