Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment

Ziyang Chen, Zhenxuan Huang, Yile Wang, Weiqin Wang, Lu Yin, Hui Huang

TL;DR

SemPA introduces semantic preference alignment for large language models to improve sentence embeddings while preserving generative capabilities. By formulating paraphrase generation as a sentence-level DPO task and linking DPO to contrastive learning under the Plackett-Luce framework, SemPA achieves robust semantic representations with lightweight fine-tuning (LoRA) on LLaMA backbones. Empirically, it yields strong STS performance, better embedding space uniformity/isotropy, and maintains broad generative abilities across tasks, outperforming prompt-based baselines and several embedding models under data-efficient regimes. The work provides a theoretical bridge between contrastive learning and preference optimization and highlights practical gains for semantically rich representations in generative LLMs.

Abstract

Traditional sentence embedding methods employ token-level contrastive learning on non-generative pre-trained models. Recently, there have emerged embedding methods based on generative large language models (LLMs). These methods either rely on fixed prompt templates or involve modifications to the model architecture. The former lacks further optimization of the model and results in limited performance, while the latter alters the internal computational mechanisms of the model, thereby compromising its generative capabilities. We propose SemPA, a novel approach that boosts the sentence representations while preserving the generative ability of LLMs via semantic preference alignment. We leverage sentence-level Direct Preference Optimization (DPO) to efficiently optimize LLMs on a paraphrase generation task, where the model learns to discriminate semantically equivalent sentences while preserving inherent generative capacity. Theoretically, we establish a formal connection between DPO and contrastive learning under the Plackett-Luce model framework. Empirically, experimental results on both semantic textual similarity tasks and various benchmarks for LLMs show that SemPA achieves better semantic representations without sacrificing the inherent generation capability of LLMs.

SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment

TL;DR

SemPA introduces semantic preference alignment for large language models to improve sentence embeddings while preserving generative capabilities. By formulating paraphrase generation as a sentence-level DPO task and linking DPO to contrastive learning under the Plackett-Luce framework, SemPA achieves robust semantic representations with lightweight fine-tuning (LoRA) on LLaMA backbones. Empirically, it yields strong STS performance, better embedding space uniformity/isotropy, and maintains broad generative abilities across tasks, outperforming prompt-based baselines and several embedding models under data-efficient regimes. The work provides a theoretical bridge between contrastive learning and preference optimization and highlights practical gains for semantically rich representations in generative LLMs.

Abstract

Traditional sentence embedding methods employ token-level contrastive learning on non-generative pre-trained models. Recently, there have emerged embedding methods based on generative large language models (LLMs). These methods either rely on fixed prompt templates or involve modifications to the model architecture. The former lacks further optimization of the model and results in limited performance, while the latter alters the internal computational mechanisms of the model, thereby compromising its generative capabilities. We propose SemPA, a novel approach that boosts the sentence representations while preserving the generative ability of LLMs via semantic preference alignment. We leverage sentence-level Direct Preference Optimization (DPO) to efficiently optimize LLMs on a paraphrase generation task, where the model learns to discriminate semantically equivalent sentences while preserving inherent generative capacity. Theoretically, we establish a formal connection between DPO and contrastive learning under the Plackett-Luce model framework. Empirically, experimental results on both semantic textual similarity tasks and various benchmarks for LLMs show that SemPA achieves better semantic representations without sacrificing the inherent generation capability of LLMs.
Paper Structure (19 sections, 12 equations, 4 figures, 7 tables)

This paper contains 19 sections, 12 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Our Method
  5. Paraphrase Generation Task
  6. Semantic Preference Alignment via DPO
  7. Acquisition of Sentence Embedding
  8. Unifying Contrastive Learning and DPO
  9. Experiments
  10. Settings
  11. Main Results
  12. Analyses
  13. Conclusion
  14. More Implementation Details
  15. Uniformity and Isotropy Score
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Comparison of sentence embedding methods. Top: contrastive learning for encoder-only models. Bottom: our semantic preference alignment for LLMs.
  • Figure 2: The overall pipeline of our proposed SemPA method. (a) We first construct the paraphrase generation preference pairs using NLI datasets (§\ref{['subsec:paraphrase']}). (b) Then we perform semantic DPO training on LLMs (§\ref{['subsec:dpotraining']}). (c) The final sentence embedding is acquired using the PromptEOL template (§\ref{['subsec:acquisition']}).
  • Figure 3: Performance of LLaMA models on the STS-B dev set and STS test sets with different training data sizes. STS test results are averaged over all STS tasks.
  • Figure 4: Qualitative visualization of embedding space using LLaMA2 (left) and LLaMA3 (right). Our method results in a more isotropic embedding space.