House of Cards: Massive Weights in LLMs
Jaehoon Oh, Seungjun Shin, Dokwan Oh
TL;DR
Massive activations bias LLMs by concentrating large magnitudes in a small set of feature dimensions in early FFN layers. The authors show these effects originate from the top-$k$ rows of ${\bm{W}}_{up}$ (and ${\bm{W}}_{gate}$) that map the FFN intermediate state $\hat{h}^{inter}_{l}$, revealing learning is dominated by a tiny weight subset during pre-training. They propose MacDrop, a curriculum dropout that targets these massive weights during parameter-efficient fine-tuning to reduce dependency on pre-trained massive weights. Across zero-shot, long-context, and ablation experiments, MacDrop improves performance and robustness for many models, though it is not uniformly effective across all architectures (e.g., Phi-3-medium and Gemma-2). These findings reveal a weight-space structure in LLMs and offer a practical, plug-and-play tool to bolster PEFT robustness.
Abstract
Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the corresponding token. In this paper, we identify that massive activations originate not from the hidden state but from the intermediate state of a feed-forward network module in an early layer. Expanding on the previous observation that massive activations occur only in specific feature dimensions, we dive deep into the weights that cause massive activations. Specifically, we define top-$k$ massive weights as the weights that contribute to the dimensions with the top-$k$ magnitudes in the intermediate state. When these massive weights are set to zero, the functionality of LLMs is entirely disrupted. However, when all weights except for massive weights are set to zero, it results in a relatively minor performance drop, even though a much larger number of weights are set to zero. This implies that during the pre-training process, learning is dominantly focused on massive weights. Building on this observation, we propose a simple plug-and-play method called MacDrop (massive weights curriculum dropout), to rely less on massive weights during parameter-efficient fine-tuning. This method applies dropout to the pre-trained massive weights, starting with a high dropout probability and gradually decreasing it as fine-tuning progresses. Through various experiments, including zero-shot downstream tasks, long-context tasks, and ablation studies, we demonstrate that \texttt{MacDrop} generally improves performance and strengthens robustness.