Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Learning Demonstration Selection via Influence Analysis

Vinay M. S., Minh-Hao Van, Xintao Wu

TL;DR

This work tackles the problem of selecting demonstrations for in-context learning in large language models by introducing InfICL, an influence-analysis-based method that identifies the most impactful training samples using embeddings from a local LLM and trains a lightweight classifier to estimate per-sample influence. Demonstrations are constructed by selecting the top-R influential samples per class, enabling balanced, high-quality prompts without fine-tuning the external LLM. A running-cost analysis shows InfICL requires mainly local-LLM embeddings and a single external inference per test, offering significant cost advantages over retraining-based influence methods. Empirical evaluations on CoLA, RTE, and SST2 with multiple external LLMs demonstrate that InfICL often achieves superior ICL performance while reducing costly API calls, though some dataset-model combinations may favor alternative strategies; the work also analyzes influence-score ranges and statistical significance to support the approach.

Abstract

Large Language Models (LLMs) have showcased their In-Context Learning (ICL) capabilities, enabling few-shot learning without the need for gradient updates. Despite its advantages, the effectiveness of ICL heavily depends on the choice of demonstrations. Selecting the most effective demonstrations for ICL remains a significant research challenge. To tackle this issue, we propose a demonstration selection method named InfICL, which utilizes influence functions to analyze impacts of training samples. By identifying the most influential training samples as demonstrations, InfICL aims to enhance the ICL generalization performance. To keep InfICL cost-effective, we only use the LLM to generate sample input embeddings, avoiding expensive fine-tuning. Through empirical studies on various real-world datasets, we demonstrate advantages of InfICL compared to state-of-the-art baselines.

In-Context Learning Demonstration Selection via Influence Analysis

TL;DR

This work tackles the problem of selecting demonstrations for in-context learning in large language models by introducing InfICL, an influence-analysis-based method that identifies the most impactful training samples using embeddings from a local LLM and trains a lightweight classifier to estimate per-sample influence. Demonstrations are constructed by selecting the top-R influential samples per class, enabling balanced, high-quality prompts without fine-tuning the external LLM. A running-cost analysis shows InfICL requires mainly local-LLM embeddings and a single external inference per test, offering significant cost advantages over retraining-based influence methods. Empirical evaluations on CoLA, RTE, and SST2 with multiple external LLMs demonstrate that InfICL often achieves superior ICL performance while reducing costly API calls, though some dataset-model combinations may favor alternative strategies; the work also analyzes influence-score ranges and statistical significance to support the approach.

Abstract

Large Language Models (LLMs) have showcased their In-Context Learning (ICL) capabilities, enabling few-shot learning without the need for gradient updates. Despite its advantages, the effectiveness of ICL heavily depends on the choice of demonstrations. Selecting the most effective demonstrations for ICL remains a significant research challenge. To tackle this issue, we propose a demonstration selection method named InfICL, which utilizes influence functions to analyze impacts of training samples. By identifying the most influential training samples as demonstrations, InfICL aims to enhance the ICL generalization performance. To keep InfICL cost-effective, we only use the LLM to generate sample input embeddings, avoiding expensive fine-tuning. Through empirical studies on various real-world datasets, we demonstrate advantages of InfICL compared to state-of-the-art baselines.
Paper Structure (11 sections, 5 equations, 1 figure, 6 tables, 1 algorithm)

This paper contains 11 sections, 5 equations, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Algorithm
  5. Running Cost Analysis
  6. Design Intuitions
  7. Experiments
  8. Experimental Setup
  9. Experimental Results
  10. Conclusion
  11. Limitations

Figures (1)

  • Figure 1: Illustration of ICL for the text classification task through our InfICL. Initially, by employing the local LLM, embeddings for all the training and validation set inputs are generated. A local classifier is then trained by employing training input embeddings and labels. InfICL determines $K$ demonstration examples based on influence scores. Finally, the demonstration set and each test case are sent to an external LLM for inference.