D.Va: Validate Your Demonstration First Before You Use It

TL;DR

This work proposes a novel method, D.Va demonstration validation mechanism, which effectively identifies demonstrations that are both effective and highly generalizable and surpasses all existing demonstration selection techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks.

Abstract

In-context learning (ICL) has demonstrated significant potential in enhancing the capabilities of large language models (LLMs) during inference. It's well-established that ICL heavily relies on selecting effective demonstrations to generate outputs that better align with the expected results. As for demonstration selection, previous approaches have typically relied on intuitive metrics to evaluate the effectiveness of demonstrations, which often results in limited robustness and poor cross-model generalization capabilities. To tackle these challenges, we propose a novel method, \textbf{D}emonstration \textbf{VA}lidation (\textbf{D.Va}), which integrates a demonstration validation perspective into this field. By introducing the demonstration validation mechanism, our method effectively identifies demonstrations that are both effective and highly generalizable. \textbf{D.Va} surpasses all existing demonstration selection techniques across both natural language understanding (NLU) and natural language generation (NLG) tasks. Additionally, we demonstrate the robustness and generalizability of our approach across various language models with different retrieval models.

Figures (6)

• Figure 1: Collaborative comparison of the average perplexity, performance, and cross-model performance of different methods across eight NLU datasets on Llama-3.2-1B. Cross-model refers to selecting demonstrations with Llama-3.2-1B while inferring with Llama-3.1-8B. Although MDL and ConE outperform the data-dependent baseline TopK in terms of performance, they don't effectively reduce the model's perplexity on the ground-truth labels and show limited cross-model generalization capabilities.
• Figure 2: The main framework of D.Va. We first retrieve the nearest demonstration as the validation example and a demonstration candidate set of size $K-1$. Then use our proposed metric to re-rank all the candidates and concatenate the top $n$ candidates as the final context at the inference stage.
• Figure 3: (a) The performance of our method compared to other methods on GPT2-XL, Llama-3.2-1B, Llama-3.2-3B and Llama-3.1-8B, respectively. (b) The performance of various methods using different numbers of in-context examples on Llama-3.2-1B. (c) The overall performance of our method across eight NLU datasets using different values of $\lambda$ on Llama-3.2-1B.
• Figure 4: The performance of our method on Trec and SQuAD v2 using different values of $\lambda$ on Llama-3.2-1B (top) and Llama-3.1-8B (bottom).
• Figure 5: Impact of the number of candidates retrieved by the TopK method. The amount of $\clubsuit$ refers to the real-time costs under the same value of $K$.