Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass

Bowen Zhang, Zixin Song, Chunping Li

TL;DR

CSE-SFP tackles the challenge of unsupervised sentence representation with large generative PLMs by leveraging a two-stage prompt to produce two distinct embeddings in a single forward pass, enabling efficient contrastive learning. The method demonstrates strong gains in semantic similarity and retrieval tasks while substantially reducing training time and memory usage. It introduces ratio-based alignment and uniformity metrics to better evaluate the semantic space and analyzes how the approach mitigates anisotropy and over-smoothing. Overall, CSE-SFP offers a practical and scalable path for deploying unsupervised sentence representations with decoder-only LLMs in IR/NLP applications.

Abstract

As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.

CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass

TL;DR

CSE-SFP tackles the challenge of unsupervised sentence representation with large generative PLMs by leveraging a two-stage prompt to produce two distinct embeddings in a single forward pass, enabling efficient contrastive learning. The method demonstrates strong gains in semantic similarity and retrieval tasks while substantially reducing training time and memory usage. It introduces ratio-based alignment and uniformity metrics to better evaluate the semantic space and analyzes how the approach mitigates anisotropy and over-smoothing. Overall, CSE-SFP offers a practical and scalable path for deploying unsupervised sentence representations with decoder-only LLMs in IR/NLP applications.

Abstract

As a fundamental task in Information Retrieval and Computational Linguistics, sentence representation has profound implications for a wide range of practical applications such as text clustering, content analysis, question-answering systems, and web search. Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field, particularly through unsupervised embedding derivation methods centered on discriminative PLMs like BERT. However, due to time and computational constraints, few efforts have attempted to integrate unsupervised sentence representation with generative PLMs, which typically possess much larger parameter sizes. Given that state-of-the-art models in both academia and industry are predominantly based on generative architectures, there is a pressing need for an efficient unsupervised text representation framework tailored to decoder-only PLMs. To address this concern, we propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models. Compared to existing strategies, CSE-SFP requires only a single forward pass to perform effective unsupervised contrastive learning. Rigorous experimentation demonstrates that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption. Furthermore, we introduce two ratio metrics that jointly assess alignment and uniformity, thereby providing a more robust means for evaluating the semantic spatial properties of encoding models.
Paper Structure (17 sections, 12 equations, 2 figures, 7 tables)

This paper contains 17 sections, 12 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Task Definition
  4. Contrastive Learning for Sentence Embedding
  5. Constructing Positive Examples
  6. Methodology
  7. Motivation
  8. CSE-SFP
  9. Experiments
  10. Implementation Details
  11. Performance on Semantic Textual Similarity Tasks
  12. Performance on Information Retrieval Tasks
  13. Computational Cost Comparison
  14. Analysis
  15. Alignment and Uniformity
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Workflow comparison between traditional methods (e.g., SimCSE) and CSE-SFP. SimCSE generates positive samples via built-in dropout within the Transformer block, requiring an additional copy of the same text and performing two forward computations to acquire the anchor sentence embedding $h_i$ and its positive counterpart $h_i^+$. In contrast, CSE-SFP concatenates two distinct manual templates, allowing both embeddings to be generated in a single forward pass.
  • Figure 2: The overall architecture of CSE-SFP. By taking full advantage of LLMs' structural as well as functional characteristics, we obtain $h_m^i$ and $h_n^i$ for constructing positive sample pairs in contrastive learning with just a single forward pass. Moreover, both the prefix and suffix of CSE-SFP are flexible, allowing for customization based on different PLMs and downstream tasks. Here, we exemplify the assembly of a two-stage prompt using PromptSTH and PromptSUM, as proposed by PretCoTandKE PretCoTandKE-ICIC-2024.