Exploring the Role of Diversity in Example Selection for In-Context Learning

TL;DR

The paper addresses the problem that in-context learning (ICL) performance depends on the chosen demonstrations, and similarity-only retrieval can introduce topical bias that caps gains. It introduces Diversity-based In-Context Learning (DICL), which reranks a large candidate pool with Maximum Marginal Relevance to balance relevance to the input with inter-example diversity, without requiring additional training. Empirical results across multiple datasets, LLM families, and similarity functions show that DICL yields consistent downstream improvements, with larger context sizes ($k$) and moderate diversity weights ($\alpha$) often yielding the strongest gains. The work demonstrates a scalable, training-free approach to enhance ICL robustness and generalization, with practical implications for prompting in real-world applications.

Abstract

In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized examples relevant to a given input using sparse or dense embedding-based similarity functions. However, relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance. We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance. To achieve this, we leverage maximum marginal relevance (MMR) which balances topical similarity with inter-example diversity. Our experimental results demonstrate that diversifying the selected examples leads to consistent improvements in downstream performance across various context sizes and similarity functions. The implementation of our approach is made available at https://github.com/janak11111/Diverse-ICL.

Figures (2)

• Figure 1: A schematic representation of Diversity-based In-Context Learning (DICL). The figure illustrates intermediate steps in the topic diversification process for a sample input (a sample movie review from the SST2 dataset). In this example, a total of $k=8$ candidates need to be selected. We schematically illustrate the selection mechanism employed during three different steps of the algorithm with two different diversity preference settings for MMR. The bottom row of steps corresponds to the case when the relative importance of diversity is higher than the train-test similarity. It can be seen that this leads to a large number of examples being selected from the set $\mathcal{N}_K - \mathcal{N}_k$, i.e., from the larger pool of of size $K=nk$ of candidate examples that do not belong to the interior of the local $k$-neighborhood (the circle) and are relatively farther away from the test query.
• Figure 2: Effect of varying the number of examples in ICL and DICL across 4 datasets and 3 LLMs.