Small Singular Values Matter: A Random Matrix Analysis of Transformer Models

TL;DR

The paper investigates how information is distributed across the singular spectrum of transformer weight matrices by applying Random Matrix Theory as a baseline and analyzing deviations from the Marchenko-Pastur law in Bert, Pythia, and Llama. It reveals that not only large outliers but also small singular values carry meaningful directions, evidenced by overlaps with activation covariance eigenvectors and by significant performance sensitivity when pruning these values. A minimal random-matrix model explains how nontrivial noise allows small-end outliers to emerge and relate to learning quality. These findings have practical implications for SVD-based pruning and model compression, and they emphasize the need to consider the entire spectrum, including the often-neglected small singular values, during fine-tuning and pruning.

Abstract

This work analyzes singular-value spectra of weight matrices in pretrained transformer models to understand how information is stored at both ends of the spectrum. Using Random Matrix Theory (RMT) as a zero information hypothesis, we associate agreement with RMT as evidence of randomness and deviations as evidence for learning. Surprisingly, we observe pronounced departures from RMT not only among the largest singular values -- the usual outliers -- but also among the smallest ones. A comparison of the associated singular vectors with the eigenvectors of the activation covariance matrices shows that there is considerable overlap wherever RMT is violated. Thus, significant directions in the data are captured by small singular values and their vectors as well as by the large ones. We confirm this empirically: zeroing out the singular values that deviate from RMT raises language-model perplexity far more than removing values from the bulk, and after fine-tuning the smallest decile can be the third most influential part of the spectrum. To explain how vectors linked to small singular values can carry more information than those linked to larger values, we propose a linear random-matrix model. Our findings highlight the overlooked importance of the low end of the spectrum and provide theoretical and practical guidance for SVD-based pruning and compression of large language models.

Figures (15)

• Figure 1: Spectra of random matrices (a,c) and trained matrices (b,d) in comparison to the theoretically predicted Marchenko curve for square matrices (a-b) and non-square matrices (c-d). Panel (a) shows that the theoretical prediction for a square matrix with dimension N=768 agrees perfectly with the empirical spectrum of a untrained weight matrix. Panel (b) shows that training led to the emergence of outliers in the Key matrix of the fifth block from a Pythia model. Panel (c) shows the perfect agreement of theory and initialized weight matrices in the case of non-square matrices. Square and non-square matrices differ fundamentally, as non-square matrices can also exhibit outliers to the left, as displayed in panel (d) for the Up-Projection matrix of block 5 for a Pythia model.
• Figure 2: Spectra and maximum overlap $O_k$ of the corresponding right singular vectors with eigenvectors of the activation covariance matrix (see Eq. \ref{['Eq:maxOverlap']}) for square matrices of different LLMs. We observe that outliers of the Marchenko-Pastur bulk have a significantly increased overlap, indicating that important information might be carried. In the case of square matrices, singular values can only exit the bulk in the region of large singular values.
• Figure 3: Spectra and maximum overlap $O_k$ of the corresponding right singular vectors with eigenvectors of the activation covariance matrix for rectangular matrices of different LLMs. The singular vectors corresponding to large singular values exhibit a significant overlap with the activation covariance matrix. However, in the case of non-square matrices, we also observe an increased overlap with the singular vectors corresponding to the smallest singular values. This is surprising as the smallest singular values contribute very little to the variance of the activations in the next layer and are typically regarded as something that can be discarded.
• Figure 4: Spectra and the overlap $O_k$ for block 10 of Pythia and Llama. We find that in the case of non-square matrices, the singular value outliers to the left of the spectrum can have a significantly increased overlap $O_k$ with the eigenvectors of the activation covariance matrix. This is the case for the Up-Projection matrix of Pythia and Llama, as well as Llama's Gate-Projection matrix. In case of the Attention-Output matrices and the Down-Projection matrix of Llama, we find very little overlap. This is a systematic finding and may reflect the training dynamics (see Appendix for details).
• Figure 5: Increase in perplexity on WikiText for Pythia and Llama when removing deciles of rank-ordered singular values. Singular value deciles are removed from all blocks, but only from a specific matrix type, e.g., all Key matrices. The inset shows the respective spectra averaged over all blocks. As expected, removing the largest singular values substantially affects perplexity for both models and all matrix types, as the matrix changes significantly upon the removal. Interestingly, we find for non-square matrices, i.e., matrices with spectra that have outliers towards small values, that the decile with the smallest singular values is more important than some of the larger deciles. In case of the Down-Projection matrix, these singular values are even the second most important group. These findings confirm that meaningful information may reside in the lower end of the spectrum.