Effective In-Context Example Selection through Data Compression

TL;DR

The paper tackles the problem of selecting informative in-context demonstrations for in-context learning (ICL) by proposing a two-stage data compression framework. It first recalls a broad set of candidate examples using BM25, then re-ranks recalled examples with a meta-gradient–based influence score, leveraging a Fisher-information–based Hessian approximation to avoid second-order derivatives. The method yields consistent improvements over a BM25 baseline across five real-world datasets and multiple GPT-2 models, with average gains around $5.90\%$. This data-independent approach compresses essential information from the training set into the in-context prompts, potentially enhancing the robustness and efficiency of ICL in practical settings.

Abstract

In-context learning has been extensively validated in large language models. However, the mechanism and selection strategy for in-context example selection, which is a crucial ingredient in this approach, lacks systematic and in-depth research. In this paper, we propose a data compression approach to the selection of in-context examples. We introduce a two-stage method that can effectively choose relevant examples and retain sufficient information about the training dataset within the in-context examples. Our method shows a significant improvement of an average of 5.90% across five different real-world datasets using four language models.