From Attention to Activation: Unravelling the Enigmas of Large Language Models

TL;DR

These methods reduce the attention proportion on the first token from 65% to 3.3%, the activation kurtosis in the hidden states from 1657 to 3.1, and perplexity penalty under 4-bit weight quantisation from 3565 to 0.3.

Abstract

We study two strange phenomena in auto-regressive Transformers: (1) the dominance of the first token in attention heads; (2) the occurrence of large outlier activations in the hidden states. We find that popular large language models, such as Llama attend maximally to the first token in 98% of attention heads, a behaviour we attribute to the softmax function. To mitigate this issue, we propose a reformulation of softmax to softmax-1. Furthermore, we identify adaptive optimisers, e.g. Adam, as the primary contributor to the large outlier activations and introduce OrthoAdam, a novel optimiser that utilises orthogonal matrices to transform gradients, to address this issue. Finally, not only do our methods prevent these phenomena from occurring, but additionally, they enable Transformers to sustain their performance when quantised using basic algorithms, something that standard methods are unable to do. In summary, our methods reduce the attention proportion on the first token from 65% to 3.3%, the activation kurtosis in the hidden states from 1657 to 3.1, and perplexity penalty under 4-bit weight quantisation from 3565 to 0.3.

Figures (42)

• Figure 1:
• Figure 2:
• Figure 4: Relaxing causal masking leads to attention domination by a different token
• Figure 5: Rotating vectors with dominant components leads to a reduction in the maximum absolute value.
• Figure 6: Layer-wise progression of first token attention dominance for GPT2-60M. The x-axis is normalised to the range $[0, 1]$. S1/OA denote models trained with softmax-1 and/or OrthoAdam.