Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models

TL;DR

The paper tackles the challenge of deploying large LLaMA-based LLMs by quantizing activations while mitigating architecture-specific activation spikes. It critically examines the universality of outlier behavior, revealing that spikes in LLaMA-like models concentrate in a small set of projection layers and propagate through residual paths, with BOT token spikes playing a notable role. The authors propose a spike-aware mixed-precision PTQ strategy that uses FP8 for spike-prone projections and lower-precision quantization elsewhere, achieving superior perplexity and zero-shot accuracy, especially at 8-bit per-tensor quantization, across LLaMA2, LLaMA3, and Mistral. This architecture-specific approach outperforms general-purpose outlier-handling methods and offers a practical path toward efficient LLM deployment in resource-constrained environments, while leaving room for further improvements and integration with complementary quantization techniques.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, their size presents significant challenges for deployment and inference. This paper investigates the quantization of LLMs, focusing on the LLaMA architecture and its derivatives. We challenge existing assumptions about activation outliers in LLMs and propose a novel mixed-precision quantization approach tailored for LLaMA-like models. Our method leverages the observation that activation spikes in LLaMA architectures are predominantly concentrated in specific projection layers. By applying higher precision (FP16 or FP8) to these layers while quantizing the rest of the model to lower bit-widths, we achieve superior performance compared to existing quantization techniques. Experimental results on LLaMA2, LLaMA3, and Mistral models demonstrate significant improvements in perplexity and zero-shot accuracy, particularly for 8-bit per-tensor quantization. Our approach outperforms general-purpose methods designed to handle outliers across all architecture types, highlighting the benefits of architecture-specific quantization strategies. This research contributes to the ongoing efforts to make LLMs more efficient and deployable, potentially enabling their use in resource-constrained environments. Our findings emphasize the importance of considering model-specific characteristics in developing effective quantization pipelines for state-of-the-art language models by identifying and targeting a small number of projections that concentrate activation spikes.

Figures (6)

• Figure 1: Causal architecture of OPT and LLaMA model. There are 2 main differences : the location of the layer norm and RMS Norm, and the MLP in LLaMA uses a gate projection
• Figure 2: Layer 19 activations values of (a) the first RMSNorm's input in a LLaMA3-8B (b) k-proj output in a Mistral 7B
• Figure 3: Maximum absolute value over layers for a LLaMA3-8B. Each color represent a different projection and we clearly see that down_proj has the biggest spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model.
• Figure 4: Maximum absolute value over layers for a LLaMA2-7B.
• Figure 5: Maximum absolute value over layers for a LLaMA2-13B.