Effective Demonstration Annotation for In-Context Learning via Language Model-Based Determinantal Point Process

TL;DR

This work tackles the challenge of relying on large labeled support sets for in-context learning by proposing LM-DPP, a selective annotation framework that jointly optimizes for low uncertainty and high diversity in demonstrations. It computes a perplexity-based score $r_i$ to quantify uncertainty and builds a diversity-aware kernel within a conditional DPP, balanced by a parameter $\lambda$ to form $L'$, enabling efficient greedy MAP inference. The method demonstrates consistent improvements across 9 NLU and 2 Generation tasks and shows strong transferability across model scales, while remaining computationally efficient through single-pass scoring and cached perplexities. The study highlights practical gains in ICL performance with limited annotation budgets and provides insight into the trade-offs between uncertainty and diversity, with implications for resource-constrained deployment of large language models. Overall, LM-DPP offers a principled, scalable strategy for annotation-efficient ICL, with demonstrated robustness across models and tasks and clear directions for future work in broader language coverage and annotation-free approaches.

Abstract

In-context learning (ICL) is a few-shot learning paradigm that involves learning mappings through input-output pairs and appropriately applying them to new instances. Despite the remarkable ICL capabilities demonstrated by Large Language Models (LLMs), existing works are highly dependent on large-scale labeled support sets, not always feasible in practical scenarios. To refine this approach, we focus primarily on an innovative selective annotation mechanism, which precedes the standard demonstration retrieval. We introduce the Language Model-based Determinant Point Process (LM-DPP) that simultaneously considers the uncertainty and diversity of unlabeled instances for optimal selection. Consequently, this yields a subset for annotation that strikes a trade-off between the two factors. We apply LM-DPP to various language models, including GPT-J, LlaMA, and GPT-3. Experimental results on 9 NLU and 2 Generation datasets demonstrate that LM-DPP can effectively select canonical examples. Further analysis reveals that LLMs benefit most significantly from subsets that are both low uncertainty and high diversity.

Figures (8)

• Figure 1: Left (Step 1): Without assuming access to a large amount of labeled data, we employ active data collection, selectively annotating demonstration examples. Right (Step 2): Prompt construction and model inference.
• Figure 2: An illustration of our proposed approach. There are three steps in LM-DPP: (1) Estimate the perplexity for each unlabeled data point, with the reciprocal denoted as $r(x_i)$. (2) Employ conditional DPP to jointly model uncertainty and diversity, selecting a small set of examples for annotation before test time. (3) At test time, the context is constructed by retrieving relevant examples from the small annotated pool.
• Figure 3: LlaMA-2-7B Results with $\mathcal{L} = 4$.
• Figure 4: Comparisons of various selection methods with ({16, 100, 300, 800}) annotated examples on four representative tasks: RTE, MRPC paraphrase detection, QNLI, and Hellaswag commonsense answering for GPT-J.
• Figure 5: Results of GPT-3-Turbo (175B) with 100 annotated examples. LM-DPP consistently improves in-context learning on various datasets.