HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives

TL;DR

This work proposes HNCSE, a novel contrastive learning framework that extends the leading SimCSE approach that makes innovative use of hard negative samples to enhance the learning of both positive and negative samples, thereby achieving a deeper semantic understanding.

Abstract

Unsupervised sentence representation learning remains a critical challenge in modern natural language processing (NLP) research. Recently, contrastive learning techniques have achieved significant success in addressing this issue by effectively capturing textual semantics. Many such approaches prioritize the optimization using negative samples. In fields such as computer vision, hard negative samples (samples that are close to the decision boundary and thus more difficult to distinguish) have been shown to enhance representation learning. However, adapting hard negatives to contrastive sentence learning is complex due to the intricate syntactic and semantic details of text. To address this problem, we propose HNCSE, a novel contrastive learning framework that extends the leading SimCSE approach. The hallmark of HNCSE is its innovative use of hard negative samples to enhance the learning of both positive and negative samples, thereby achieving a deeper semantic understanding. Empirical tests on semantic textual similarity and transfer task datasets validate the superiority of HNCSE.

Figures (4)

• Figure 1: An example of a query with its positive, over-easy, over-hard and missing hard negative samples. The figure shows the situation that two too similar sentences may cause the loss of normal hard negative samples after passing through the encoder.
• Figure 2: Schematic diagram of Positive Mixing.
• Figure 3: Schematic diagram of Hard Negative Mixing.
• Figure 4: The influence of different hyperparameters on HNCSE.