Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering

Zifeng Cheng, Zhonghui Wang, Yuchen Fu, Zhiwei Jiang, Yafeng Yin, Cong Wang, Qing Gu

TL;DR

This work proposes a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding and can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information.

Abstract

Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP.

Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering

TL;DR

This work proposes a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding and can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information.

Abstract

Extracting sentence embeddings from large language models (LLMs) is a practical direction, as it requires neither additional data nor fine-tuning. Previous studies usually focus on prompt engineering to guide LLMs to encode the core semantic information of the sentence into the embedding of the last token. However, the last token in these methods still encodes an excess of non-essential information, such as stop words, limiting its encoding capacity. To this end, we propose a Contrastive Prompting (CP) method that introduces an extra auxiliary prompt to elicit better sentence embedding. By contrasting with the auxiliary prompt, CP can steer existing prompts to encode the core semantics of the sentence, rather than non-essential information. CP is a plug-and-play inference-time intervention method that can be combined with various prompt-based methods. Extensive experiments on Semantic Textual Similarity (STS) tasks and downstream classification tasks demonstrate that our method can improve the performance of existing prompt-based methods across different LLMs. Our code will be released at https://github.com/zifengcheng/CP.
Paper Structure (24 sections, 8 equations, 3 figures, 13 tables)

This paper contains 24 sections, 8 equations, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Extracting Embeddings from LLMs
  5. Multi-Head Attention in LLMs
  6. Method
  7. Auxiliary Prompt
  8. Contrastive Activation Steering
  9. Norm Adjustment
  10. Intermediate Embedding Eliciting
  11. Experiments
  12. Datasets and Experimental Settings
  13. Baselines
  14. Results
  15. Effects of Different Backbones
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The comparison of sentence embeddings obtained by the auxiliary prompt, normal prompt, and our proposed inference-time steering method. The decoding probabilities of Next Token Prediction are used to reflect the semantic information contained in the corresponding sentence embeddings.
  • Figure 2: Illustration of the contrastive prompting method.
  • Figure 3: Effects of intervention layer, scaling factor, and output layer. (a) Effects of the intervention layer on norm scaling. (b) Effects of the intervention layer on norm recovering. (c) Effects of the scaling factor on norm scaling. (d) The effects of the output layer on Knowledge+CP-NS.