Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

Rahul Atul Bhope, Praveen Venkateswaran, K. R. Jayaram, Vatche Isahagian, Vinod Muthusamy, Nalini Venkatasubramanian

TL;DR

The paper addresses fragility in in-context learning caused by the order of in-context examples. It introduces OptiSeq, an inference-time, dataset-free method that exhaustively evaluates all permutations of examples to identify the best order by maximizing output log-probability after removing the examples, thereby improving distinguishability between outputs. A variant, EOptiSeq, prunes the search using SBERT-based similarities to anchor promising examples and reduce computational burden. Across API sequence generation and text classification tasks on diverse models and datasets, OptiSeq achieves notable gains over random, Top-K, and other baselines, demonstrating the practical value of on-the-fly example ordering for robust ICL.

Abstract

Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks.

OptiSeq: Ordering Examples On-The-Fly for In-Context Learning

TL;DR

The paper addresses fragility in in-context learning caused by the order of in-context examples. It introduces OptiSeq, an inference-time, dataset-free method that exhaustively evaluates all permutations of examples to identify the best order by maximizing output log-probability after removing the examples, thereby improving distinguishability between outputs. A variant, EOptiSeq, prunes the search using SBERT-based similarities to anchor promising examples and reduce computational burden. Across API sequence generation and text classification tasks on diverse models and datasets, OptiSeq achieves notable gains over random, Top-K, and other baselines, demonstrating the practical value of on-the-fly example ordering for robust ICL.

Abstract

Developers using LLMs and LLM-based agents in their applications have provided plenty of anecdotal evidence that in-context-learning (ICL) is fragile. In this paper, we show that in addition to the quantity and quality of examples, the order in which the in-context examples are listed in the prompt affects the output of the LLM and, consequently, their performance. While prior work has explored improving ICL through dataset-dependent techniques, we introduce OptiSeq, a purely inference-time, dataset-free optimization method that efficiently determines the best example order. OptiSeq leverages log probabilities of LLM-generated outputs to systematically prune the search space of possible orderings and recommend the best order(s) by distinguishing orderings that yield high levels of accuracy and those that underperform. Extensive empirical evaluation on multiple LLMs, datasets, and prompts demonstrate that OptiSeq improves accuracy by 5.5 - 10.5 percentage points across multiple tasks.
Paper Structure (23 sections, 2 equations, 12 figures, 13 tables, 1 algorithm)

This paper contains 23 sections, 2 equations, 12 figures, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Sensitivity of ICL to Example Ordering
  3. Naive ICL fails to distinguish between correct and incorrect outputs.
  4. Optimal order varies across instances and models.
  5. Impact of choosing the wrong order can be significant depending on the dataset or task.
  6. Methodology
  7. OptiSeq
  8. EOptiSeq
  9. Evaluation
  10. Datasets
  11. Models
  12. Comparative Techniques
  13. Metrics
  14. Results and Discussions
  15. Related Work
  16. ...and 8 more sections

Figures (12)

  • Figure 1: llama-3-8b-instruct performance variation with 6 in-context example orderings using 3 examples.
  • Figure 2: Naive ICL exhibits higher overlap in log probabilities for correct and incorrect outpus, making distinction harder.
  • Figure 3: AG News classification accuracy across different orders and models
  • Figure 4: Variation in the average accuracy across all permutations of three examples for five datasets
  • Figure 5: OptiSeq Overview
  • ...and 7 more figures