Whitening Not Recommended for Classification Tasks in LLMs
Ali Forooghi, Shaghayegh Sadeghi, Jianguo Lu
TL;DR
The paper systematically evaluates whitening as a post-processing technique for sentence embeddings from large language models, covering PCA, ZCA, PCA-Cor, ZCA-Cor, and Cholesky whitenings. It centers embeddings with $X-\mu$, decorrelates via the covariance $\Sigma = (X-\mu)(X-\mu)^T$, and applies a whitening matrix $W$ so that the transformed covariance is the identity $I$, e.g., PCA whitening with $W = U \Lambda^{-1/2} U^T$. Across eight models and seven datasets, whitening consistently degrades classification performance, while its impact on STS tasks is model-dependent; some LLMs show mild gains, but others do not. The isotropy-focused IsoScore analysis reveals that whitening increases isotropy, yet higher isotropy does not guarantee better downstream performance, with notable improvements in some vanilla models and limited or negative effects in fine-tuned ones like SBERT/SimCSE and ChatGPT. As a by-product, the work introduces SentEval$^+$, an embedding evaluation platform enabling affordable experimentation on commodity hardware, aiding practitioners and researchers to assess embedding quality across models and tasks.
Abstract
Sentence embedding is a cornerstone in NLP. Whitening has been claimed to be an effective operation to improve embedding quality obtained from Large Language Models (LLMs). However, we find that the efficacy of whitening is model-dependent and task-dependent. In particular, whitening degenerates embeddings for classification tasks. The conclusion is supported by extensive experiments. We also explored a variety of whitening operations, including PCA, ZCA, PCA-Cor, ZCA-Cor and Cholesky whitenings. A by-product of our research is embedding evaluation platform for LLMs called SentEval+.