$Se^2$: Sequential Example Selection for In-Context Learning

TL;DR

This work tackles the sensitivity of in-context learning (ICL) to the choice of demonstration examples. It introduces Se^2, a sequential-aware framework that learns to construct high-quality example sequences by leveraging feedback from LLMs and using beam search to expand the search space. The method combines a scoring mechanism (for NLU and NLG), a context-sequence construction process, a contrastive training objective, and an efficient inference pipeline, and is validated across 23 NLP tasks with substantial gains over random and competitive baselines. The results demonstrate strong stability and cross-model transferability, underscoring the practical impact of modeling inter-example relationships in prompts; code is available for replication at the authors’ repository.

Abstract

The remarkable capability of large language models (LLMs) for in-context learning (ICL) needs to be activated by demonstration examples. Prior work has extensively explored the selection of examples for ICL, predominantly following the "select then organize" paradigm, such approaches often neglect the internal relationships between examples and exist an inconsistency between the training and inference. In this paper, we formulate the problem as a $Se$quential $Se$lection problem and introduce $Se^2$, a sequential-aware method that leverages the LLM's feedback on varying context, aiding in capturing inter-relationships and sequential information among examples, significantly enriching the contextuality and relevance of ICL prompts. Meanwhile, we utilize beam search to seek and construct example sequences, enhancing both quality and diversity. Extensive experiments across 23 NLP tasks from 8 distinct categories illustrate that $Se^2$ markedly surpasses competitive baselines and achieves 42\% relative improvement over random selection. Further in-depth analysis shows the effectiveness of proposed strategies, highlighting $Se^2$'s exceptional stability and adaptability across various scenarios. Code available at https://github.com/microsoft/LMOps.

Figures (5)

• Figure 1: The "select then organize" paradigm and our sequential example selection.
• Figure 2: The process of Context Sequence Construction, mainly includes sampling examples, scoring and ranking examples for varying context input, and selecting and extending example sequences.
• Figure 3: The process of searching for example sequences during inference using beam search. We draw the case of $w=1$ to illustrate this briefly.
• Figure 4: The performance distribution of $Se^2$ and random sampling when tuning in-context prompts.
• Figure 5: Two case studies on ARC-C where $Se^2$ helps LLM infer the correct answer, but UPRISE does not.