Mitigating Outlier Activations in Low-Precision Fine-Tuning of Language Models

TL;DR

This work tackles the vulnerability of low-precision language-model fine-tuning to outlier activations that distort scaling and hinder small-value representation. It proposes a comprehensive integer-based pipeline using dynamic fixed-point (block floating-point) mapping to represent most data in $INT8$ while isolating outliers in $INT12$ or tiling them across two $INT8$ streams, controlled by a fixed threshold $\gamma=5$. The authors provide a theoretical framework linking information loss to quantization variance and model outliers as a mixture distribution, arguing that separating outliers preserves informativeness. Empirical results on BERT-base fine-tuning for GLUE and SQuAD demonstrate improved robustness and competitive performance versus FP16 baselines, with all forward computations conducted in low-precision. This approach offers a practical path to cost- and energy-efficient low-precision fine-tuning suitable for resource-constrained settings, while preserving gradient accuracy through an $INT8$-based training pipeline.

Abstract

Low-precision fine-tuning of language models has gained prominence as a cost-effective and energy-efficient approach to deploying large-scale models in various applications. However, this approach is susceptible to the existence of outlier values in activation. The outlier values in the activation can negatively affect the performance of fine-tuning language models in the low-precision regime since they affect the scaling factor and thus make representing smaller values harder. This paper investigates techniques for mitigating outlier activation in low-precision integer fine-tuning of the language models. Our proposed novel approach enables us to represent the outlier activation values in 8-bit integers instead of floating-point (FP16) values. The benefit of using integers for outlier values is that it enables us to use operator tiling to avoid performing 16-bit integer matrix multiplication to address this problem effectively. We provide theoretical analysis and supporting experiments to demonstrate the effectiveness of our approach in improving the robustness and performance of low-precision fine-tuned language models.

Figures (3)

• Figure 1: Computation flow of proposed linear layers for forward and backward propagation. Integer computation significantly reduces the computational cost of compute-intensive linear layers.
• Figure 2: Inference operations in an integer-only linear layer. The bottom panel shows the linear fixed-point mapping for the input tensors, which can adapt different bit-widths for activation outliers and other parameters in the layer.
• Figure 3: Outliers modeled as a mixture distribution.