Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reordering Examples Helps during Priming-based Few-Shot Learning

Sawan Kumar, Partha Talukdar

TL;DR

PERO reframes few-shot learning as a permutation search problem over training examples, using a genetic algorithm to find the optimal order (and optionally a separator token) for prompting pretrained language models. It demonstrates that example order is a critical factor for generalization, enabling strong performance with as few as 10 examples across sentiment analysis, natural language inference, and fact retrieval, while remaining competitive with existing baselines as data grows. The work provides ablations and analyses of learned permutations, offering practical one-shot learning recipes and insights into prompt design for low-data regimes. By treating prompting as an interpretable probe rather than parameter updates, PERO advances data-efficient evaluation and deployment of large pretrained models.

Abstract

The ability to learn from limited data, or few-shot learning, is a desirable and often critical requirement for NLP systems. While many existing methods do poorly at learning from a handful of examples, large pretrained language models have recently been shown to be efficient few-shot learners. One approach to few-shot learning, which does not require finetuning of model parameters, is to augment the language model's input with priming text which is typically constructed using task specific descriptions and examples. In this work, we further explore priming-based few-shot learning, with focus on using examples as prompts. We show that presenting examples in the right order is key for generalization. We introduce PERO (Prompting with Examples in the Right Order), where we formulate few-shot learning as search over the set of permutations of the training examples. We show that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. While the newline token is a natural choice for separating the examples in the prompt, we show that learning a new separator token can potentially provide further gains in performance. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. Finally, we analyze the learned prompts to reveal novel insights, including the idea that two training examples in the right order alone can provide competitive performance for sentiment classification and natural language inference.

Reordering Examples Helps during Priming-based Few-Shot Learning

TL;DR

PERO reframes few-shot learning as a permutation search problem over training examples, using a genetic algorithm to find the optimal order (and optionally a separator token) for prompting pretrained language models. It demonstrates that example order is a critical factor for generalization, enabling strong performance with as few as 10 examples across sentiment analysis, natural language inference, and fact retrieval, while remaining competitive with existing baselines as data grows. The work provides ablations and analyses of learned permutations, offering practical one-shot learning recipes and insights into prompt design for low-data regimes. By treating prompting as an interpretable probe rather than parameter updates, PERO advances data-efficient evaluation and deployment of large pretrained models.

Abstract

The ability to learn from limited data, or few-shot learning, is a desirable and often critical requirement for NLP systems. While many existing methods do poorly at learning from a handful of examples, large pretrained language models have recently been shown to be efficient few-shot learners. One approach to few-shot learning, which does not require finetuning of model parameters, is to augment the language model's input with priming text which is typically constructed using task specific descriptions and examples. In this work, we further explore priming-based few-shot learning, with focus on using examples as prompts. We show that presenting examples in the right order is key for generalization. We introduce PERO (Prompting with Examples in the Right Order), where we formulate few-shot learning as search over the set of permutations of the training examples. We show that PERO can learn to generalize efficiently using as few as 10 examples, in contrast to existing approaches. While the newline token is a natural choice for separating the examples in the prompt, we show that learning a new separator token can potentially provide further gains in performance. We demonstrate the effectiveness of the proposed method on the tasks of sentiment classification, natural language inference and fact retrieval. Finally, we analyze the learned prompts to reveal novel insights, including the idea that two training examples in the right order alone can provide competitive performance for sentiment classification and natural language inference.

Paper Structure

This paper contains 38 sections, 1 figure, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background: Genetic Algorithm
  4. PERO: Proposed Method
  5. Notation and Input Format
  6. Genetic Algorithm: Search over Permutations of Examples
  7. Separator Token Learning
  8. Experiments
  9. Datasets
  10. Sentiment Classification:
  11. NLI:
  12. Fact Retrieval:
  13. Experimental Setup
  14. Number of training examples:
  15. Pretrained LM:
  16. ...and 23 more sections

Figures (1)

  • Figure 1: Overview of PERO: Given a set of training examples, PERO searches over permutations of training examples using a genetic algorithm (Section \ref{['subsec:method_search']}), and optionally also learns a separator token (Section \ref{['subsec:method_separator']}) which is used to concatenate the training examples. Briefly, starting from a set of randomly initialized permutations, the genetic algorithm step computes the fitness of each permutation for making predictions using a pretrained LM. These fitness scores are then used for selection and subsequent breeding of new permutations using biologically inspired operations of mutation and crossover. The separator token learning step uses the updated set of permutations and uses gradient updates to improve the separator token. The two steps are performed iteratively for a fixed number of epochs and the best permutation and separator token are selected using a validation set. Please see Section \ref{['sec:method']} for details.