DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning

TL;DR

DenoSent tackles the limitation of contrastive-only sentence representation learning by introducing an intra-sentence denoising objective. It perturbs sentences via discrete (back-translation/LLMs) and continuous (dropout) noises and trains a Transformer-based autoencoder to reconstruct the original from its noisy input, using a pooled encoder representation for denoising. The framework can be integrated with contrastive learning to provide both intra- and inter-sentence supervision, yielding competitive results on STS and improvements across reranking, retrieval, and classification tasks. Empirical results, supported by ablations, demonstrate the complementary nature of the denoising objective and its robustness across domains, with the authors releasing their code for replication.

Abstract

Contrastive-learning-based methods have dominated sentence representation learning. These methods regularize the representation space by pulling similar sentence representations closer and pushing away the dissimilar ones and have been proven effective in various NLP tasks, e.g., semantic textual similarity (STS) tasks. However, it is challenging for these methods to learn fine-grained semantics as they only learn from the inter-sentence perspective, i.e., their supervision signal comes from the relationship between data samples. In this work, we propose a novel denoising objective that inherits from another perspective, i.e., the intra-sentence perspective. By introducing both discrete and continuous noise, we generate noisy sentences and then train our model to restore them to their original form. Our empirical evaluations demonstrate that this approach delivers competitive results on both semantic textual similarity (STS) and a wide range of transfer tasks, standing up well in comparison to contrastive-learning-based methods. Notably, the proposed intra-sentence denoising objective complements existing inter-sentence contrastive methodologies and can be integrated with them to further enhance performance. Our code is available at https://github.com/xinghaow99/DenoSent.

Figures (4)

• Figure 1: Overview of DenoSent. The proposed sentence representation learning framework is a combination of two objectives, providing both inter-sentence and intra-sentence supervision signals. Note that we use pooling strategies to downsize the encoder outputs from [n_tokens, hidden_dim] to [1, hidden_dim].
• Figure 2: The two-stage perturbation process wherein both discrete and continuous noises are sequentially incorporated into the original sentences. The discrete perturbation is achieved through back-translation or the use of a large language model (LLM), while the continuous perturbation is implemented by applying substantial dropout on the embedded sentences.
• Figure 3: Absolute performance difference on reranking and retrieval tasks compared to SimCSE. AUD, MS, SciD, SODQ and QR denotes AskUbuntuDupQuestions, MindSmallReranking, SciDocsRR, StackOverflowDupQuestions and QuoraRetrieval, respectively.
• Figure 4: Average STS performance using different numbers of attention heads and dropout rates.