MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learning

TL;DR

MAPLE addresses the data bottleneck in many-shot in-context learning by using influence-based selection to pseudo-label informative unlabeled samples and by adaptively choosing demonstrations per query. It constructs a labeled-unlabeled graph to identify high-impact unlabeled samples and forms a high-quality demonstration pool, then tailors demonstrations to each test input via a secondary pseudo-labeled graph. The approach yields consistent improvements across eight real-world tasks and shows robustness across different LLMs, while analyzing the trade-offs with demonstration quantity and order. Practically, MAPLE reduces labeling costs while delivering strong performance in long-context ICL scenarios.

Abstract

In-Context Learning (ICL) empowers Large Language Models (LLMs) to tackle diverse tasks by incorporating multiple input-output examples, known as demonstrations, into the input of LLMs. More recently, advancements in the expanded context windows of LLMs have led to many-shot ICL, which uses hundreds of demonstrations and outperforms few-shot ICL, which relies on fewer examples. However, this approach is often hindered by the high cost of obtaining large amounts of labeled data. To address this challenge, we propose Many-Shot Adaptive Pseudo-LabEling, namely MAPLE, a novel influence-based many-shot ICL framework that utilizes pseudo-labeled samples to compensate for the lack of label information. We first identify a subset of impactful unlabeled samples and perform pseudo-labeling on them by querying LLMs. These pseudo-labeled samples are then adaptively selected and tailored to each test query as input to improve the performance of many-shot ICL, without significant labeling costs. Extensive experiments on real-world datasets demonstrate the effectiveness of our framework, showcasing its ability to enhance LLM adaptability and performance with limited labeled data.

Figures (6)

• Figure 1: Accuracies on Date and GPQA datasets with different amount of demonstrations. The LLM is Gemini 1.5 Flash.
• Figure 2: Overview of the MAPLE framework. Given a dataset with a small fraction of labeled samples, we select unlabeled samples for pseudo-labeling using the proposed influence score. During inference, we adopt a similar approach to select relevant samples from the candidate pool for each query, filtering out unrelated ones. The remaining samples are then used for many-shot ICL.
• Figure 3: Performance comparison of various sample selection strategies in a many-shot ICL setting using Gemini 1.5 Flash across multiple datasets. 'Zero-shot' refers to the query LLM being provided only with the task instruction. We randomly selected 20 labeled samples to construct $\mathcal{D}_L$, and the results obtained using just $\mathcal{D}_L$ are presented as 'Few-shot'. Based on $\mathcal{D}_L$, we compare MAPLE with other pseudo-labeling baselines with different $\mathcal{D}_U^{*}$.
• Figure 4: Results of MAPLE compared to baselines on two datasets with a larger number of demonstrations. We increase the size of $\mathcal{D}_L$ to 100 and compare performance for different sizes of $\mathcal{D}_U^{*}$ (50, 100, 150, and 200).
• Figure 5: The results of varying the fraction of pseudo-labeled demonstrations. We fix the size of the candidate demonstration pool $\mathcal{D}_F$ as 120 and adjust the proportion of pseudo-labeled samples, while randomly selecting the labeled samples.

Theorems & Definitions (5)

• Definition 3.1
• Theorem 3.2
• Lemma 1.1
• proof
• proof