Skill-Based Few-Shot Selection for In-Context Learning

TL;DR

Skill-KNN introduces a training-free, skill-based approach to few-shot selection for in-context learning by prompting a frozen LLM to rewrite inputs into skill-focused descriptions and then retrieving similar exemplars via embeddings. This rewrite-then-retrieve pipeline aims to mitigate surface-form biases inherent in raw-input-based selections and to accommodate dynamic, expanding example banks without fine-tuning. Across five cross-domain semantic-parsing datasets and six backbones, Skill-KNN consistently outperforms raw-input methods and is often competitive with, or superior to, fine-tuning-based selectors; its two variants further improve robustness and diversity. The work demonstrates that extracting intrinsic task-specific skills through prompting can significantly enhance cross-domain generalization in in-context learning, albeit with substantial GPU resource requirements and potential applicability beyond semantic parsing for future exploration.

Abstract

In-context learning is the paradigm that adapts large language models to downstream tasks by providing a few examples. Few-shot selection -- selecting appropriate examples for each test instance separately -- is important for in-context learning. In this paper, we propose Skill-KNN, a skill-based few-shot selection method for in-context learning. The key advantages of Skill-KNN include: (1) it addresses the problem that existing methods based on pre-trained embeddings can be easily biased by surface natural language features that are not important for the target task; (2) it does not require training or fine-tuning of any models, making it suitable for frequently expanding or changing example banks. The key insight is to optimize the inputs fed into the embedding model, rather than tuning the model itself. Technically, Skill-KNN generates the skill-based descriptions for each test case and candidate example by utilizing a pre-processing few-shot prompting, thus eliminating unimportant surface features. Experimental results across five cross-domain semantic parsing datasets and six backbone models show that Skill-KNN significantly outperforms existing methods.

Figures (8)

• Figure 1: In-context learning with different selection methods. (a) Examples from raw-input-based selection just share similar entities with the input query. (b) With the skill-based description, the selected examples contain the desired task-specific skills.
• Figure 2: The bird's-eye view of Skill-KNN, a rewrite-then-retrieve selection method to facilitate in-context learning with skill-based descriptions.
• Figure 3: Two variants of Skill-KNN. The blue and green points represent two candidate sets of skill-based representations.
• Figure 4: Performance of Skill-KNN (base version) with different number of annotated demonstrations.
• Figure 5: Performance of Skill-KNN (base version) with constraints on selecting examples for annotating.