Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning

Javad Rafiei Asl, Prajwal Panzade, Eduardo Blanco, Daniel Takabi, Zhipeng Cai

TL;DR

This paper introduces RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks.

Abstract

Pre-trained language models (PLMs) have consistently demonstrated outstanding performance across a diverse spectrum of natural language processing tasks. Nevertheless, despite their success with unseen data, current PLM-based representations often exhibit poor robustness in adversarial settings. In this paper, we introduce RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks. Through the generation of high-risk adversarial perturbations and their utilization in a novel objective function, RobustSentEmbed adeptly learns high-quality and robust sentence embeddings. Our experiments confirm the superiority of RobustSentEmbed over state-of-the-art representations. Specifically, Our framework achieves a significant reduction in the success rate of various adversarial attacks, notably reducing the BERTAttack success rate by almost half (from 75.51\% to 38.81\%). The framework also yields improvements of 1.59\% and 0.23\% in semantic textual similarity tasks and various transfer tasks, respectively.

RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning

TL;DR

This paper introduces RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks.

Abstract

Pre-trained language models (PLMs) have consistently demonstrated outstanding performance across a diverse spectrum of natural language processing tasks. Nevertheless, despite their success with unseen data, current PLM-based representations often exhibit poor robustness in adversarial settings. In this paper, we introduce RobustSentEmbed, a self-supervised sentence embedding framework designed to improve both generalization and robustness in diverse text representation tasks and against a diverse set of adversarial attacks. Through the generation of high-risk adversarial perturbations and their utilization in a novel objective function, RobustSentEmbed adeptly learns high-quality and robust sentence embeddings. Our experiments confirm the superiority of RobustSentEmbed over state-of-the-art representations. Specifically, Our framework achieves a significant reduction in the success rate of various adversarial attacks, notably reducing the BERTAttack success rate by almost half (from 75.51\% to 38.81\%). The framework also yields improvements of 1.59\% and 0.23\% in semantic textual similarity tasks and various transfer tasks, respectively.
Paper Structure (22 sections, 14 equations, 6 figures, 6 tables, 1 algorithm)

This paper contains 22 sections, 14 equations, 6 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Proposed Framework
  4. Perturbation Generator
  5. Robust Contrastive Learning
  6. Evaluation and Experimental Results
  7. Adversarial Attacks
  8. Robust Embeddings
  9. Semantic Textual Similarity (STS) Tasks
  10. Transfer Tasks
  11. Distribution of Sentence Embeddings
  12. Conclusion and Future Work
  13. Limitations
  14. Training Details
  15. Ablation Studies
  16. ...and 7 more sections

Figures (6)

  • Figure 1: The general architecture of the RobustSentEmbed framework.
  • Figure 2: Average number of queries and the resulting accuracy reduction for two fine-tuned embeddings.
  • Figure 3: $\ell_\text{align}-\ell_\text{uniform}$ plot of models based on e. Lower uniformity and alignment is better.
  • Figure 4: The impact of step sizes in perturbation generation on the average performance of STS tasks.
  • Figure 5: The impact of the step number (represented by N = K or T) in the T-step FGSM and K-step PGD methods on the averaged correlation of the STS tasks.
  • ...and 1 more figures