Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation

Jian Qian, Miao Sun, Sifan Zhou, Ziyu Zhao, Ruizhi Hun, Patrick Chiang

TL;DR

In Sub-SA, a submodular function is designed that facilitates effective subset selection for annotation and RPR (Reward and Penalty Regularization) is proposed to better balance the diversity and representativeness of the unlabeled dataset attributed to a reward term and a penalty term.

Abstract

In-context learning (ICL) leverages in-context examples as prompts for the predictions of Large Language Models (LLMs). These prompts play a crucial role in achieving strong performance. However, the selection of suitable prompts from a large pool of labeled examples often entails significant annotation costs. To address this challenge, we propose Sub-SA (Submodular Selective Annotation), a submodule-based selective annotation method. The aim of Sub-SA is to reduce annotation costs while improving the quality of in-context examples and minimizing the time consumption of the selection process. In Sub-SA, we design a submodular function that facilitates effective subset selection for annotation and demonstrates the characteristics of monotonically and submodularity from the theoretical perspective. Specifically, we propose RPR (Reward and Penalty Regularization) to better balance the diversity and representativeness of the unlabeled dataset attributed to a reward term and a penalty term, respectively. Consequently, the selection for annotations can be effectively addressed with a simple yet effective greedy search algorithm based on the submodular function. Finally, we apply the similarity prompt retrieval to get the examples for ICL.

Sub-SA: Strengthen In-context Learning via Submodular Selective Annotation

TL;DR

In Sub-SA, a submodular function is designed that facilitates effective subset selection for annotation and RPR (Reward and Penalty Regularization) is proposed to better balance the diversity and representativeness of the unlabeled dataset attributed to a reward term and a penalty term.

Abstract

In-context learning (ICL) leverages in-context examples as prompts for the predictions of Large Language Models (LLMs). These prompts play a crucial role in achieving strong performance. However, the selection of suitable prompts from a large pool of labeled examples often entails significant annotation costs. To address this challenge, we propose Sub-SA (Submodular Selective Annotation), a submodule-based selective annotation method. The aim of Sub-SA is to reduce annotation costs while improving the quality of in-context examples and minimizing the time consumption of the selection process. In Sub-SA, we design a submodular function that facilitates effective subset selection for annotation and demonstrates the characteristics of monotonically and submodularity from the theoretical perspective. Specifically, we propose RPR (Reward and Penalty Regularization) to better balance the diversity and representativeness of the unlabeled dataset attributed to a reward term and a penalty term, respectively. Consequently, the selection for annotations can be effectively addressed with a simple yet effective greedy search algorithm based on the submodular function. Finally, we apply the similarity prompt retrieval to get the examples for ICL.
Paper Structure (19 sections, 1 theorem, 15 equations, 9 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 1 theorem, 15 equations, 9 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Preliminaries
  2. In-context Learning
  3. Submodular Subset Selection
  4. Selective annotation Problem Setting
  5. Methodology
  6. Selective Annotation
  7. Reward and Penalty Regularization
  8. Submodular Function
  9. Greedy Search Algorithm
  10. Prompt Retrieval
  11. Experiments
  12. Experimental Setups
  13. Implementation Details
  14. Main Results
  15. Analysis
  16. ...and 4 more sections

Key Result

Theorem 1

$A$ denotes the solution obtained by the greedy search approach, and $A^*$ denotes the optimal solution. If $\mathcal{F}(\cdot)$ is a submodular function, then the solution $A$ has the following approximation guarantee: where $\mathrm{e}$ is the base of natural logarithm.

Figures (9)

  • Figure 1: Comparison on performance and time consumption during subset selection under the same hardware condition. The $y$-axis represents the performance of different datasets on the classification task, and the $x$-axis represents the time consumption with the log scale. Here the annotation budget is 18. Our Sub-SA can remarkably outperform the Vote-$k$ baseline by a large average margin ($5.0\%$ absolute gain and $2.9\times$ acceleration of millisecond-level log representation). Notably, Sub-SA outperforms the IDEAL on the SST-2 benchmark with an improvement of $12.6\%$ absolute gain.
  • Figure 2: Comparison on different methods for selective annotation.
  • Figure 3: The selective annotation pipeline for in-context learning. Given a pool of unlabeled instances $\mathcal{U}$, the goal of selective annotation is to select the most informative examples $\mathcal{A}$ to annotate based on a fixed budget $T$.Then, retrieve the annotated set based on unseen test data to get the $k$ shot examples, and finally concatenate the test data and $k$ in-context examples to maximize ICL performance with frozen LLM.
  • Figure : a. Annotation Budget is 100
  • Figure : a. MNLI
  • ...and 4 more figures

Theorems & Definitions (2)

  • Definition 1: Submodular Function edmonds2003submodular
  • Theorem 1