Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, Samuel R. Bowman

TL;DR

<3-5 sentence high-level summary> Systematic, large-scale evaluation compares 19 pretraining tasks and transfer paradigms for sentence-level encoders, evaluating on GLUE with standardized architectures. The study finds language-model pretraining to be the most reliable signal, with modest gains from intermediate training and multitask learning, but substantial variability across target tasks and signs of catastrophic forgetting in some setups. It reveals weak cross-task correlations, suggesting no single pretraining objective benefits all targets, and highlights the need for more sophisticated transfer-learning methods beyond simple multitask aggregation. Overall, the work clarifies the practical limits of current multitask/transfer approaches and underscores scaling language modeling as the most straightforward path to improvements.

Abstract

Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling. We conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks both as alternatives and complements to language modeling. Our primary results support the use language modeling, especially when combined with pretraining on additional labeled-data tasks. However, our results are mixed across pretraining tasks and show some concerning trends: In ELMo's pretrain-then-freeze paradigm, random baselines are worryingly strong and results vary strikingly across target tasks. In addition, fine-tuning BERT on an intermediate task often negatively impacts downstream transfer. In a more positive trend, we see modest gains from multitask training, suggesting the development of more sophisticated multitask and transfer learning techniques as an avenue for further research.

Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

TL;DR

<3-5 sentence high-level summary> Systematic, large-scale evaluation compares 19 pretraining tasks and transfer paradigms for sentence-level encoders, evaluating on GLUE with standardized architectures. The study finds language-model pretraining to be the most reliable signal, with modest gains from intermediate training and multitask learning, but substantial variability across target tasks and signs of catastrophic forgetting in some setups. It reveals weak cross-task correlations, suggesting no single pretraining objective benefits all targets, and highlights the need for more sophisticated transfer-learning methods beyond simple multitask aggregation. Overall, the work clarifies the practical limits of current multitask/transfer approaches and underscores scaling language modeling as the most straightforward path to improvements.

Abstract

Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling. We conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks both as alternatives and complements to language modeling. Our primary results support the use language modeling, especially when combined with pretraining on additional labeled-data tasks. However, our results are mixed across pretraining tasks and show some concerning trends: In ELMo's pretrain-then-freeze paradigm, random baselines are worryingly strong and results vary strikingly across target tasks. In addition, fine-tuning BERT on an intermediate task often negatively impacts downstream transfer. In a more positive trend, we see modest gains from multitask training, suggesting the development of more sophisticated multitask and transfer learning techniques as an avenue for further research.

Paper Structure

This paper contains 44 sections, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transfer Paradigms
  4. Pretraining
  5. Intermediate Training
  6. Target Task Evaluation
  7. Tasks
  8. Random Encoder
  9. GLUE Tasks
  10. Outside Tasks
  11. Multitask Learning
  12. Models and Experimental Details
  13. Encoder Architecture
  14. Task-Specific Components
  15. Training and Optimization
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Learning settings that we consider. Model components with frozen parameters are shown in gray and decorated with snowflakes. Top (pretraining): We pretrain a BiLSTM on a task (left), and learn a target task model on top of the representations it produces (right). Middle (intermediate ELMo training): We train a BiLSTM on top of ELMo for an intermediate task (left). We then train a target task model on top of the intermediate task BiLSTM and ELMo (right). Bottom (intermediate BERT training): We fine-tune BERT on an intermediate task (left), and then fine-tune the resulting model again on a target task (right).
  • Figure 2: Learning curves (log scale) showing overall GLUE scores for encoders pretrained to convergence with varying amounts of data, shown for pretraining (left) and intermediate ELMo (center) and BERT (right) training.
  • Figure 3: Target-task training learning curves for each GLUE task with three encoders: the random encoder without ELMo (left), random with ELMo (center), and MTL Non-GLUE pretraining (right).