Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

Shufan Wang, Laure Thompson, Mohit Iyyer

TL;DR

Phrase-BERT tackles the weakness of standard BERT phrase embeddings by introducing dual contrastive fine-tuning on lexically diverse paraphrases and phrase-context data. By combining a paraphrase-based objective with context-aware training, Phrase-BERT achieves semantically meaningful, lexically diverse phrase embeddings and improves phrase-level relatedness tasks. The approach is demonstrated not only in intrinsic semantics benchmarks but also by integrating into a phrase-based neural topic model (pntm), yielding coherent topics described by both words and phrases. The work provides practical, deployable embeddings for corpus exploration and interpretation, with publicly released code and models.

Abstract

Phrase representations derived from BERT often do not exhibit complex phrasal compositionality, as the model relies instead on lexical similarity to determine semantic relatedness. In this paper, we propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings. Our approach (Phrase-BERT) relies on a dataset of diverse phrasal paraphrases, which is automatically generated using a paraphrase generation model, as well as a large-scale dataset of phrases in context mined from the Books3 corpus. Phrase-BERT outperforms baselines across a variety of phrase-level similarity tasks, while also demonstrating increased lexical diversity between nearest neighbors in the vector space. Finally, as a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model that interprets topics as mixtures of words and phrases by performing a nearest neighbor search in the embedding space. Crowdsourced evaluations demonstrate that this phrase-based topic model produces more coherent and meaningful topics than baseline word and phrase-level topic models, further validating the utility of Phrase-BERT.

Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration

TL;DR

Phrase-BERT tackles the weakness of standard BERT phrase embeddings by introducing dual contrastive fine-tuning on lexically diverse paraphrases and phrase-context data. By combining a paraphrase-based objective with context-aware training, Phrase-BERT achieves semantically meaningful, lexically diverse phrase embeddings and improves phrase-level relatedness tasks. The approach is demonstrated not only in intrinsic semantics benchmarks but also by integrating into a phrase-based neural topic model (pntm), yielding coherent topics described by both words and phrases. The work provides practical, deployable embeddings for corpus exploration and interpretation, with publicly released code and models.

Abstract

Phrase representations derived from BERT often do not exhibit complex phrasal compositionality, as the model relies instead on lexical similarity to determine semantic relatedness. In this paper, we propose a contrastive fine-tuning objective that enables BERT to produce more powerful phrase embeddings. Our approach (Phrase-BERT) relies on a dataset of diverse phrasal paraphrases, which is automatically generated using a paraphrase generation model, as well as a large-scale dataset of phrases in context mined from the Books3 corpus. Phrase-BERT outperforms baselines across a variety of phrase-level similarity tasks, while also demonstrating increased lexical diversity between nearest neighbors in the vector space. Finally, as a case study, we show that Phrase-BERT embeddings can be easily integrated with a simple autoencoder to build a phrase-based neural topic model that interprets topics as mixtures of words and phrases by performing a nearest neighbor search in the embedding space. Crowdsourced evaluations demonstrate that this phrase-based topic model produces more coherent and meaningful topics than baseline word and phrase-level topic models, further validating the utility of Phrase-BERT.

Paper Structure

This paper contains 44 sections, 2 equations, 2 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Phrase Embeddings from BERT
  4. Using BERT to embed phrases:
  5. Creating lexically diverse phrase-level paraphrases:
  6. Collecting phrases in context:
  7. Fine-tuning BERT with a contrastive objective using the constructed datasets
  8. Implementation details:
  9. Experimental setup
  10. Datasets
  11. Turney:
  12. BiRD:
  13. PPDB:
  14. PPDB-filtered:
  15. PAWS-short:
  16. ...and 29 more sections

Figures (2)

  • Figure 1: We propose Phrase-BERT to correct BERT's embedding space for phrases by placing semantically similar phrases and contexts closer together (e.g., the anchor phrase "Jack of all trade" and the positive paraphrase "a worker with versatile skills") and by removing lexical cues with a contrastive learning objective.
  • Figure 2: Word intrusion experiments show that pntm achieves higher model precision (y-axis) than UNTM when the number of topics ($K$) is increased.