Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

Gaifan Zhang, Yi Zhou, Danushka Bollegala

TL;DR

Addressing the practicality of PLM sentence embeddings, this paper studies unsupervised post-hoc dimensionality reduction from dimension $d$ to $d'<d$ to enable memory- and compute-constrained deployments. It systematically compares PCA, SVD, Kernel PCA, Gaussian Random Projection, and Autoencoders, across six encoders and three evaluation tasks in both inductive and transductive settings. The key finding is that PCA achieves about a $50 ext{%}$ reduction in dimensionality with around a $1 ext{%}$ loss in task performance, and can even improve accuracy for some encoders; results on other methods vary by encoder and task. These findings provide practical guidance for compressing sentence embeddings in retrieval and NLU pipelines, enabling broader deployment of PLM-based representations with modest accuracy trade-offs.

Abstract

Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community due to their superior performance when representing texts in numerous downstream applications. However, the high dimensionality of the sentence embeddings produced by PLMs is problematic when representing large numbers of sentences in memory- or compute-constrained devices. As a solution, we evaluate unsupervised dimensionality reduction methods to reduce the dimensionality of sentence embeddings produced by PLMs. Our experimental results show that simple methods such as Principal Component Analysis (PCA) can reduce the dimensionality of sentence embeddings by almost $50\%$, without incurring a significant loss in performance in multiple downstream tasks. Surprisingly, reducing the dimensionality further improves performance over the original high-dimensional versions for the sentence embeddings produced by some PLMs in some tasks.

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

TL;DR

Addressing the practicality of PLM sentence embeddings, this paper studies unsupervised post-hoc dimensionality reduction from dimension to to enable memory- and compute-constrained deployments. It systematically compares PCA, SVD, Kernel PCA, Gaussian Random Projection, and Autoencoders, across six encoders and three evaluation tasks in both inductive and transductive settings. The key finding is that PCA achieves about a reduction in dimensionality with around a loss in task performance, and can even improve accuracy for some encoders; results on other methods vary by encoder and task. These findings provide practical guidance for compressing sentence embeddings in retrieval and NLU pipelines, enabling broader deployment of PLM-based representations with modest accuracy trade-offs.

Abstract

Sentence embeddings produced by Pretrained Language Models (PLMs) have received wide attention from the NLP community due to their superior performance when representing texts in numerous downstream applications. However, the high dimensionality of the sentence embeddings produced by PLMs is problematic when representing large numbers of sentences in memory- or compute-constrained devices. As a solution, we evaluate unsupervised dimensionality reduction methods to reduce the dimensionality of sentence embeddings produced by PLMs. Our experimental results show that simple methods such as Principal Component Analysis (PCA) can reduce the dimensionality of sentence embeddings by almost , without incurring a significant loss in performance in multiple downstream tasks. Surprisingly, reducing the dimensionality further improves performance over the original high-dimensional versions for the sentence embeddings produced by some PLMs in some tasks.
Paper Structure (20 sections, 4 equations, 4 figures, 1 table)

This paper contains 20 sections, 4 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Network Compression:
  4. Word Embedding Compression:
  5. Sentence Embedding Compression:
  6. Dimensionality Reduction Methods
  7. Experiments
  8. Evaluation Tasks and Datasets
  9. Results
  10. Conclusion
  11. Limitations
  12. Ethics and Broader Impact
  13. Dimensionality Reduction Methods
  14. Truncated SVD
  15. PCA
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Performance of the original sup-simcse-roberta-large sentence embeddings and its dimensionality reduced versions produced using different methods on STS-B (top), TREC (middle) and SICK-E (bottom) datasets. Results for the transductive and inductive settings are shown respectively on the left and right.
  • Figure 2: Spearman correlation coefficients on STS-B vs. the dimensionality of the sentence embeddings produced by applying different dimensionality reduction methods. Sentence embeddings are created using pre-trained stsb-bert-base (uppermost), stsb-bert-large (supper-middle), all-mpnet-base-v2 (lower-middle) and sup-simcse-roberta-large (lowermost) models. Results for the transductive and inductive settings are shown respectively on the left and right.
  • Figure 3: Classification accuracy on TREC vs. the dimensionality of the sentence embeddings produced by applying different dimensionality reduction methods. Sentence embeddings are created using pre-trained all-mpnet-base-v2 (uppermost), msmarco-roberta-base-v2 (supper-middle), paraphrase-xlm-r-multilingual-v1 (lower-middle) and sup-simcse-roberta-large (lowermost) models. Results for the transductive and inductive settings are shown respectively on the left and right.
  • Figure 4: Accuracy on SICK-E vs. the dimensionality of the sentence embeddings produced by applying different dimensionality reduction methods. Sentence embeddings are created using pre-trained all-mpnet-base-v2 (uppermost), msmarco-roberta-base-v2 (supper-middle), paraphrase-xlm-r-multilingual-v1 (lower-middle) and sup-simcse-roberta-large (lowermost) models. Results for the transductive and inductive settings are shown respectively on the left and right.