Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PromptBERT: Improving BERT Sentence Embeddings with Prompts

Ting Jiang, Jian Jiao, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Denvy Deng, Qi Zhang

TL;DR

PromptBERT rethinks BERT-based sentence embeddings by attributing poor performance to static token embedding biases and ineffective BERT layers rather than anisotropy. It introduces prompt-based sentence representations that avoid embedding biases while leveraging BERT, and pairs this with a template-denoising unsupervised contrastive objective to reduce template-induced bias. The approach yields state-of-the-art results on STS benchmarks in both unsupervised and supervised settings, and demonstrates improved stability across random seeds. This work offers a practical path to strong, bias-resilient sentence representations that exploit pre-trained language models without extensive task-specific fine-tuning.

Abstract

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.

PromptBERT: Improving BERT Sentence Embeddings with Prompts

TL;DR

PromptBERT rethinks BERT-based sentence embeddings by attributing poor performance to static token embedding biases and ineffective BERT layers rather than anisotropy. It introduces prompt-based sentence representations that avoid embedding biases while leveraging BERT, and pairs this with a template-denoising unsupervised contrastive objective to reduce template-induced bias. The approach yields state-of-the-art results on STS benchmarks in both unsupervised and supervised settings, and demonstrates improved stability across random seeds. This work offers a practical path to strong, bias-resilient sentence representations that exploit pre-trained language models without extensive task-specific fine-tuning.

Abstract

We propose PromptBERT, a novel contrastive learning method for learning better sentence representation. We firstly analyze the drawback of current sentence embedding from original BERT and find that it is mainly due to the static token embedding bias and ineffective BERT layers. Then we propose the first prompt-based sentence embeddings method and discuss two prompt representing methods and three prompt searching methods to make BERT achieve better sentence embeddings. Moreover, we propose a novel unsupervised training objective by the technology of template denoising, which substantially shortens the performance gap between the supervised and unsupervised settings. Extensive experiments show the effectiveness of our method. Compared to SimCSE, PromptBert achieves 2.29 and 2.58 points of improvement based on BERT and RoBERTa in the unsupervised setting.
Paper Structure (25 sections, 4 equations, 2 figures, 15 tables)

This paper contains 25 sections, 4 equations, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Rethinking the Sentence Embeddings of the Original BERT
  4. Prompt Based Sentence Embeddings
  5. Represent Sentence with the Prompt
  6. Prompt Search
  7. Prompt Based Contrastive Learning with Template Denoising
  8. Experiments
  9. Dataset
  10. Baselines
  11. Implementation Details
  12. Non Fine-Tuned BERT Results
  13. Fine-Tuned BERT Results
  14. Effectiveness of Prompt Based Contrastive Learning with Template Denoising
  15. Discussion
  16. ...and 10 more sections

Figures (2)

  • Figure 1: 2D visualization of token embeddings with different biases. For frequency bias, the darker the color, the higher the token frequency. For subword and case bias, yellow represents subword and red represents the token contains capital letters.
  • Figure 2: 2D visualization of static token embeddings in untying and tying weights pre-trained model. For frequency bias, the darker the color, the higher the token frequency. For subword and case bias, yellow represents subword and red represents the token contains capital letters.