Comparable Demonstrations are Important in In-Context Learning: A Novel Perspective on Demonstration Selection

TL;DR

The paper addresses demonstration bias in In-Context Learning (ICL) by arguing that a small set of demonstrations can induce input-label mappings that diverge from the task's essence. It proposes Comparable Demonstrations (CDs), constructed by minimally editing texts to flip labels, to create strong inter-demonstration comparisons and highlight the task essence, drawing on ideas from counterfactually-augmented data (CAD). Experiments with GPT-3.5-turbo on sentiment analysis (IMDb) and natural language inference (NLI) show that CDs reduce demonstration bias in instruction induction and generally improve ICL performance, with pronounced gains in out-of-distribution (OOD) settings; a hybrid strategy (CDs nearest) balances ID and OOD performance. The work offers a novel perspective on demonstration selection for ICL, demonstrating a practical path to mitigate bias and enhance generalization, while noting limitations such as the manual nature of CD creation and the focus on one-to-one demonstration relationships for future improvement.

Abstract

In-Context Learning (ICL) is an important paradigm for adapting Large Language Models (LLMs) to downstream tasks through a few demonstrations. Despite the great success of ICL, the limitation of the demonstration number may lead to demonstration bias, i.e. the input-label mapping induced by LLMs misunderstands the task's essence. Inspired by human experience, we attempt to mitigate such bias through the perspective of the inter-demonstration relationship. Specifically, we construct Comparable Demonstrations (CDs) by minimally editing the texts to flip the corresponding labels, in order to highlight the task's essence and eliminate potential spurious correlations through the inter-demonstration comparison. Through a series of experiments on CDs, we find that (1) demonstration bias does exist in LLMs, and CDs can significantly reduce such bias; (2) CDs exhibit good performance in ICL, especially in out-of-distribution scenarios. In summary, this study explores the ICL mechanisms from a novel perspective, providing a deeper insight into the demonstration selection strategy for ICL.

Figures (3)

• Figure 1: Overview of Comparable Demonstrations in ICL.
• Figure 2: Comparison of instruction quality under three demonstration selection strategies.
• Figure :