Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings

TL;DR

The paper tackles the challenge of token bias and loss of fine-grained semantics in unsupervised sentence embeddings by introducing SARCSE, which inserts an AutoEncoder-based token reconstruction stage after a pretrained language model and pairs it with a self-adaptive reconstruction loss that downweights frequent tokens. This architecture, combined with a contrastive learning objective, preserves subtle semantic differences and mitigates distribution biases in token representations. Empirical results across 7 STS tasks show that SARCSE consistently improves over SimCSE on RoBERTa-based encoders, with robustness to smaller batch sizes and a notable enhancement in alignment of sentence representations. The approach provides a plug-and-play enhancement to existing sentence encoders and highlights the importance of reconstruction-based semantics in conjunction with contrastive learning for high-quality unsupervised sentence embeddings.

Abstract

Unsupervised sentence embeddings task aims to convert sentences to semantic vector representations. Most previous works directly use the sentence representations derived from pretrained language models. However, due to the token bias in pretrained language models, the models can not capture the fine-grained semantics in sentences, which leads to poor predictions. To address this issue, we propose a novel Self-Adaptive Reconstruction Contrastive Sentence Embeddings (SARCSE) framework, which reconstructs all tokens in sentences with an AutoEncoder to help the model to preserve more fine-grained semantics during tokens aggregating. In addition, we proposed a self-adaptive reconstruction loss to alleviate the token bias towards frequency. Experimental results show that SARCSE gains significant improvements compared with the strong baseline SimCSE on the 7 STS tasks.

Figures (3)

• Figure 1: Two examples extracted from the corpus. The tokens with blue borders are the keywords in the two sentences. The deeper color of tokens means the greater importance in sentence embeddings for the two models. The importance of tokens in SimCSE is obtained from the self-attention aggregation weights of <s>. And the importance of tokens in SARCSE is obtained from the reconstruction loss of AutoEncoder. The deeper color means lower reconstruction loss. The tokens <s> and <$\backslash$s> have no loss because we do not reconstruct them. SARCSE pays more attention to fine-grained differences, but SimCSE does not, causing SimCSE to make a wrong similarity prediction between two sentences.
• Figure 2: The overall architecture of SARCSE. The deeper blue on tokens means the lower token frequency, which shows most high-frequency tokens have no determinative semantics. First, the tokens of sentences are input into the pretrained language models to get token representations. Then, the multi-scale representations are obtained using TextCNN Encoder with different convolution kernels. Integrating them with a CNN will get the sentence embeddings. Finally, we use the transposed CNN and TextCNN to reconstruct the sequence of tokens.
• Figure 3: The density plots of SimCSE and SARCSE in the test set of STS-B. The data are divided into 5 groups bu the ground truth similarity ratings. The higher ratings mean more similar. The y-axis represents the grouping situation, while the x-axis represents the cosine similarity.