Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learning

TL;DR

This paper tackles unsupervised sentence embeddings by addressing low data diversity and high data noise in LLM-synthesized samples. It introduces a knowledge-driven data augmentation pipeline that builds a knowledge graph from entities and quantities to guide LLMs in generating diverse positives and knowledge-aware negatives, paired with a Gaussian-decayed gradient-assisted contrastive objective (GCSE) that attenuates the influence of false hard negatives. The approach uses a two-stage training process with an evaluation model to filter synthesized data and a Gaussian-decayed loss term G(s_i,s'_i,τ,σ) to gradually restore gradient signals as training progresses. Experiments across multiple backbones and LLMs show state-of-the-art semantic textual similarity performance with fewer synthesized samples and smaller LLMs, highlighting the method's efficiency, robustness, andability to leverage domain-focused data for improved domain-specific sentence representations.

Abstract

Recently, using large language models (LLMs) for data augmentation has led to considerable improvements in unsupervised sentence embedding models. However, existing methods encounter two primary challenges: limited data diversity and high data noise. Current approaches often neglect fine-grained knowledge, such as entities and quantities, leading to insufficient diversity. Besides, unsupervised data frequently lacks discriminative information, and the generated synthetic samples may introduce noise. In this paper, we propose a pipeline-based data augmentation method via LLMs and introduce the Gaussian-decayed gradient-assisted Contrastive Sentence Embedding (GCSE) model to enhance unsupervised sentence embeddings. To tackle the issue of low data diversity, our pipeline utilizes knowledge graphs (KGs) to extract entities and quantities, enabling LLMs to generate more diverse samples. To address high data noise, the GCSE model uses a Gaussian-decayed function to limit the impact of false hard negative samples, enhancing the model's discriminative capability. Experimental results show that our approach achieves state-of-the-art performance in semantic textual similarity (STS) tasks, using fewer data samples and smaller LLMs, demonstrating its efficiency and robustness across various models.

Figures (12)

• Figure 1: Comparison of false positives (FP) and negatives (FN). Both the predicted scores and labels are normalized (see details in Appendix \ref{['ap_snm']}), where positives have a score greater than the label, while negatives lower than the label. False samples are identified when the root mean square error (RMSE) between the prediction and the label exceeds 0.2.
• Figure 2: The overall workflow of our method.
• Figure 3: The pipeline of knowledge extraction and data synthesis, where the solid black arrows in the Entity KG are hard edges, and dotted yellow lines are soft edges.
• Figure 4: In-batch training with Gaussian-decayed on GCSE.
• Figure 5: Spearman's correlation against the ratio of domain data to general data on the STS tasks.