Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

TL;DR

This work tackles transferring from pretrained language models for text classification without catastrophic forgetting. It proposes SiATL, which transfers a pretrained LM, appends a task-specific recurrent layer with self-attention, and trains with a joint objective $L = L_{task} + \gamma L_{LM}$ where $\gamma$ decays exponentially, plus a sequential unfreezing schedule to gradually expose pretrained layers. Across five affective and classification datasets, SiATL outperforms strong baselines such as ULMFiT, particularly in low-data regimes, while remaining end-to-end and simple to train. The approach demonstrates that a lightweight, single-step transfer-learning pipeline can match or exceed state-of-the-art results without extensive learning-rate scheduling or separate pretraining steps, with code to be released in a future version.

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

TL;DR

This work tackles transferring from pretrained language models for text classification without catastrophic forgetting. It proposes SiATL, which transfers a pretrained LM, appends a task-specific recurrent layer with self-attention, and trains with a joint objective where decays exponentially, plus a sequential unfreezing schedule to gradually expose pretrained layers. Across five affective and classification datasets, SiATL outperforms strong baselines such as ULMFiT, particularly in low-data regimes, while remaining end-to-end and simple to train. The approach demonstrates that a lightweight, single-step transfer-learning pipeline can match or exceed state-of-the-art results without extensive learning-rate scheduling or separate pretraining steps, with code to be released in a future version.

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

Paper Structure

This paper contains 9 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Model
  4. Experiments and Results
  5. Datasets
  6. Experimental Setup
  7. Results & Discussion
  8. Conclusions and Future Work
  9. Acknowledgements

Figures (3)

  • Figure 1: High-level overview of our proposed TL architecture. We transfer the pretrained LM add an extra recurrent layer and an auxiliary LM loss.
  • Figure 2: Results of our proposed approach (SiATL) (o) and ULMFiT (+) for different datasets as a function of the number of training examples.
  • Figure 3: Heatmap of the effect of $\gamma$ to $F_1$-score, evaluated on SCv2. The horizontal axis depicts the initial value of $\gamma$ and the vertical axis the final value of $\gamma$.