Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning to Search Effective Example Sequences for In-Context Learning

Xiang Gao, Ankita Sinha, Kamalika Das

TL;DR

The paper tackles the sensitivity of in-context learning performance to the choice of example sequences by proposing Beam Search-based Example Sequence Constructor (BESC). BESC jointly optimizes length, composition, arrangement, and query dependence through a prefix-suffix dual-encoder scoring model trained with contrastive learning, and uses beam search during inference to efficiently navigate the large sequence space with complexity $O(L \cdot b \cdot c) + O(N) + O(L \cdot b \cdot c \log N)$. Empirical results across six datasets and four language models show that BESC consistently outperforms baselines, with ablations confirming the importance of dynamic examples, adaptive length, arrangement, and sequential modeling. The work also demonstrates transferability of a pretrained BESC scorer and discusses limitations including applicability to open-ended tasks and potential prompting interactions and biases.

Abstract

Large language models (LLMs) demonstrate impressive few-shot learning capabilities, but their performance varies widely based on the sequence of in-context examples. Key factors influencing this include the sequence's length, composition, and arrangement, as well as its relation to the specific query. Existing methods often tackle these factors in isolation, overlooking their interdependencies. Moreover, the extensive search space for selecting optimal sequences complicates the development of a holistic approach. In this work, we introduce Beam Search-based Example Sequence Constructor (BESC), a novel method for learning to construct optimal example sequences. BESC addresses all key factors involved in sequence selection by considering them jointly during inference, while incrementally building the sequence. This design enables the use of beam search to significantly reduce the complexity of the search space. Experiments across various datasets and language models show notable improvements in performance.

Learning to Search Effective Example Sequences for In-Context Learning

TL;DR

The paper tackles the sensitivity of in-context learning performance to the choice of example sequences by proposing Beam Search-based Example Sequence Constructor (BESC). BESC jointly optimizes length, composition, arrangement, and query dependence through a prefix-suffix dual-encoder scoring model trained with contrastive learning, and uses beam search during inference to efficiently navigate the large sequence space with complexity . Empirical results across six datasets and four language models show that BESC consistently outperforms baselines, with ablations confirming the importance of dynamic examples, adaptive length, arrangement, and sequential modeling. The work also demonstrates transferability of a pretrained BESC scorer and discusses limitations including applicability to open-ended tasks and potential prompting interactions and biases.

Abstract

Large language models (LLMs) demonstrate impressive few-shot learning capabilities, but their performance varies widely based on the sequence of in-context examples. Key factors influencing this include the sequence's length, composition, and arrangement, as well as its relation to the specific query. Existing methods often tackle these factors in isolation, overlooking their interdependencies. Moreover, the extensive search space for selecting optimal sequences complicates the development of a holistic approach. In this work, we introduce Beam Search-based Example Sequence Constructor (BESC), a novel method for learning to construct optimal example sequences. BESC addresses all key factors involved in sequence selection by considering them jointly during inference, while incrementally building the sequence. This design enables the use of beam search to significantly reduce the complexity of the search space. Experiments across various datasets and language models show notable improvements in performance.

Paper Structure

This paper contains 31 sections, 2 equations, 3 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. In-Context Learning Mechanisms
  4. Example Sequence Selection
  5. Static examples
  6. Dynamic examples
  7. Example composition
  8. Example arrangement
  9. Sequence length
  10. Method
  11. Model Design
  12. Model Training
  13. Inference with Beam Search
  14. Reduced Complexity
  15. Experiments
  16. ...and 16 more sections

Figures (3)

  • Figure 1: The in-context learning (ICL) performance of large language models (LLMs) depends on the length, composition, and arrangement of example sequences. Existing methods address these factors separately, while our algorithm jointly considers them and efficiently manages the search space with beam search.
  • Figure 2: An overview of our BESC approach. (a) We train a scoring model to predict the effectiveness of an example sequence for a given query, using contrastive learning to compare LLM performance between sequences. (b) During inference, the example sequence is constructed incrementally with beam search, where the trained scorer ranks candidates and prunes nodes.
  • Figure 3: An illustration of the dual-encoder architecture of the scoring model. This design enables the model to rank the next element for a given query and prefix, allowing incremental sequence construction with beam search during inference.