Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving In-Context Few-Shot Learning via Self-Supervised Training

Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, Zornitsa Kozareva

TL;DR

The paper tackles improving in-context few-shot learning by inserting an intermediate self-supervised training stage between pretraining and downstream evaluation. It introduces four objectives—Next Sentence Generation, Masked Word Prediction, Last Phrase Prediction, and Classification—to train models in formats aligned with downstream tasks. Experiments on SuperGLUE and Natural-Instructions show that this intermediate self-supervision yields the best average performance, with benefits modulated by data amount, objective diversity, and template similarity; cross-task human supervision is complementary. The approach demonstrates a practical pathway to tailor pretraining for few-shot tasks without additional labeled data, improving instruction-following and task compliance in generated outputs.

Abstract

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.

Improving In-Context Few-Shot Learning via Self-Supervised Training

TL;DR

The paper tackles improving in-context few-shot learning by inserting an intermediate self-supervised training stage between pretraining and downstream evaluation. It introduces four objectives—Next Sentence Generation, Masked Word Prediction, Last Phrase Prediction, and Classification—to train models in formats aligned with downstream tasks. Experiments on SuperGLUE and Natural-Instructions show that this intermediate self-supervision yields the best average performance, with benefits modulated by data amount, objective diversity, and template similarity; cross-task human supervision is complementary. The approach demonstrates a practical pathway to tailor pretraining for few-shot tasks without additional labeled data, improving instruction-following and task compliance in generated outputs.

Abstract

Self-supervised pretraining has made few-shot learning possible for many NLP tasks. But the pretraining objectives are not typically adapted specifically for in-context few-shot learning. In this paper, we propose to use self-supervision in an intermediate training stage between pretraining and downstream few-shot usage with the goal to teach the model to perform in-context few shot learning. We propose and evaluate four self-supervised objectives on two benchmarks. We find that the intermediate self-supervision stage produces models that outperform strong baselines. Ablation study shows that several factors affect the downstream performance, such as the amount of training data and the diversity of the self-supervised objectives. Human-annotated cross-task supervision and self-supervision are complementary. Qualitative analysis suggests that the self-supervised-trained models are better at following task requirements.
Paper Structure (39 sections, 5 figures, 18 tables)

This paper contains 39 sections, 5 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. In-Context Few-Shot Learning.
  4. Fine-Tuning for Few-Shot Learning.
  5. Pretraining for Few-Shot Learning.
  6. Intermediate Fine-Tuning.
  7. Method
  8. Next Sentence Generation
  9. Masked Word Prediction
  10. Last Phrase Prediction
  11. Classification
  12. Experiment
  13. Training Setup
  14. Evaluation Setup
  15. Natural-Instructions.
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Examples of our self-supervised training tasks. Each example is an input-output pair constructed from the raw text.
  • Figure 2: An example of a training instance. Each instance is formed by several training examples. During training, we use left-to-right language models and compute a cross-entropy loss on the output texts (indicated by the red color in the shown example). We note that when computing the loss on the second example, the first example can be seen as task demonstrations. For brevity, we show part of the input and output texts.
  • Figure 3: Example illustrating the construction of training instances for our classification task. There are four input types, and each training instance has two or three types. As the shown instance has the following two types: "original sentences" and "different documents", it comprises examples from two different documents. The instance resembles the next sentence prediction task, encouraging models to compare topical similarities between the two examples.
  • Figure 4: Average results for the 1.3B model on SuperGLUE and Natural-Instructions when varying the number of examples used for self-supervised training.
  • Figure 5: Average results on SuperGLUE and Natural-Instructions when varying number of examples used for training.