Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks

Felix Drinkall, Janet B. Pierrehumbert, Stefan Zohren

TL;DR

The paper investigates how the dimensionality of LLM embeddings affects downstream regression under varying noise levels, using an autoencoder to compress embeddings to a latent space of dimension $d_z$ and a random forest regressor with Huber loss. Across noisy stock-return prediction and high-signal text tasks, it finds that an appropriately small $d_z$ (notably $d_z = 8$ for finance) often yields better performance, highlighting a regularisation effect from compression. The authors introduce the notion of intrinsic dimension, showing that strong predictive signals can be captured at dimensions far smaller than the original embedding, and they find that interpretable features like sentiment do not inherently outperform learned latent representations. These findings suggest adaptive dimensionality strategies for embedding-based regression and caution against assuming high-dimensional embeddings automatically improve performance in noisy settings.

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder's hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks

TL;DR

The paper investigates how the dimensionality of LLM embeddings affects downstream regression under varying noise levels, using an autoencoder to compress embeddings to a latent space of dimension and a random forest regressor with Huber loss. Across noisy stock-return prediction and high-signal text tasks, it finds that an appropriately small (notably for finance) often yields better performance, highlighting a regularisation effect from compression. The authors introduce the notion of intrinsic dimension, showing that strong predictive signals can be captured at dimensions far smaller than the original embedding, and they find that interpretable features like sentiment do not inherently outperform learned latent representations. These findings suggest adaptive dimensionality strategies for embedding-based regression and caution against assuming high-dimensional embeddings automatically improve performance in noisy settings.

Abstract

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of text can yield better performance in LLM-based regression tasks. In this paper, we compare the relative performance of embedding compression in three different signal-to-noise contexts: financial return prediction, writing quality assessment and review scoring. Our results show that compressing embeddings, in a minimally supervised manner using an autoencoder's hidden representation, can mitigate overfitting and improve performance on noisy tasks, such as financial return prediction; but that compression reduces performance on tasks that have high causal dependencies between the input and target data. Our results suggest that the success of interpretable compressed representations such as sentiment may be due to a regularising effect.

Paper Structure

This paper contains 22 sections, 7 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Datasets
  3. Stock Returns Dataset
  4. High Signal Datasets
  5. Methodology
  6. Input Processing and Chunking
  7. Dimensionality Reduction
  8. Regression Model
  9. Results
  10. Impact of Noise
  11. Conclusion
  12. Future Work
  13. Limitations
  14. Dataset Details
  15. Stock Returns Dataset
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Huber Loss on the noisy financial returns task for different autoencoder latent dimensions $d_{\text{z}}$. The performance of sentiment and emotion representations also appear inline with their respective dimensionality. The significance of each result compared to the best latent dimension is displayed using blue (p > .05), orange (p < .01) and yellow (p < .05) colours.
  • Figure 2: Huber loss averaged over Review, English Writing, and Financial Returns tasks - granular performance in Appendix \ref{['app:dataset_results']}. To display the results on one axis, the y-axis represents the error rate of each individual task as a percentage of the maximum and minimum error on that task. The performance without any compression is marked with the dashed line.
  • Figure 3: The normalised Huber loss of each dataset that makes up the result in Figure \ref{['fig:noise-comp']}. "raw" appears in the same location as 768 in the plot since this is the dimension of the non-compressed embedding.
  • Figure 4: Cosine similarity between $\mathbf{v}_i$ and $\hat{\mathbf{v}}_i$ on the financial returns dataset.