Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CERT: Contrastive Self-supervised Learning for Language Understanding

Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, Pengtao Xie

TL;DR

CERT presents a sentence-level contrastive self-supervised pretraining method that uses back-translation to create augmented sentence pairs and MoCo-based contrastive learning to align these pairs. By finetuning on GLUE tasks, CERT achieves higher averaged performance than BERT and shows notable gains on data-scarce tasks. The approach is flexible, data-efficient, and can be integrated with various pretrained language models, highlighting the value of sentence-level semantically-aware objectives. Overall, CERT demonstrates that contrastive sentence-level pretraining can enhance language understanding with relatively modest computational resources.

Abstract

Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT

CERT: Contrastive Self-supervised Learning for Language Understanding

TL;DR

CERT presents a sentence-level contrastive self-supervised pretraining method that uses back-translation to create augmented sentence pairs and MoCo-based contrastive learning to align these pairs. By finetuning on GLUE tasks, CERT achieves higher averaged performance than BERT and shows notable gains on data-scarce tasks. The approach is flexible, data-efficient, and can be integrated with various pretrained language models, highlighting the value of sentence-level semantically-aware objectives. Overall, CERT demonstrates that contrastive sentence-level pretraining can enhance language understanding with relatively modest computational resources.

Abstract

Pretrained language models such as BERT, GPT have shown great effectiveness in language understanding. The auxiliary predictive tasks in existing pretraining approaches are mostly defined on tokens, thus may not be able to capture sentence-level semantics very well. To address this issue, we propose CERT: Contrastive self-supervised Encoder Representations from Transformers, which pretrains language representation models using contrastive self-supervised learning at the sentence level. CERT creates augmentations of original sentences using back-translation. Then it finetunes a pretrained language encoder (e.g., BERT) by predicting whether two augmented sentences originate from the same sentence. CERT is simple to use and can be flexibly plugged into any pretraining-finetuning NLP pipeline. We evaluate CERT on 11 natural language understanding tasks in the GLUE benchmark where CERT outperforms BERT on 7 tasks, achieves the same performance as BERT on 2 tasks, and performs worse than BERT on 2 tasks. On the averaged score of the 11 tasks, CERT outperforms BERT. The data and code are available at https://github.com/UCSD-AI4H/CERT

Paper Structure

This paper contains 18 sections, 2 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Pretraining of Transformers for Language Understanding
  3. Transformer
  4. BERT
  5. Contrastive Self-supervised Learning
  6. CERT
  7. Data Augmentation
  8. CSSL Pretraining
  9. Experiments
  10. Tasks and Datasets
  11. Experimental Settings
  12. Results
  13. Ablation on Data Augmentation
  14. Ablation on CSSL Pretraining Corpora
  15. Related Works
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Keys in the queue are encoded using a momentum encoder. Given an augmented data example in the current minibatch (called query) and a key in the queue, they are considered as a positive pair if they originate from the same data example, and a negative pair if otherwise. A similarity score is calculated between the encoding of the query and the encoding of each key. Contrastive losses are defined on the similarity scores and binary labels.
  • Figure 2: The workflow of CERT. Given the large-scale input texts (without labels) from source tasks, a BERT model is first pretrained on these texts. Then we continue to train this pretrained BERT model using CSSL on the input texts (without labels) from the target task. We refer to this model as pretrained CERT model. Then we finetune the CERT model using the input texts and their associated labels in the target task and get the final model that performs the target task.
  • Figure 3: The workflow of data augmentation based on back translation.