Fine-Tuned In-Context Learners for Efficient Adaptation

TL;DR

The paper tackles efficient adaptation of large language models under limited ground-truth data by proposing a unified approach that fine-tunes on $k$-shot in-context prompts (ICL+FT). It combines the sample efficiency of in-context learning with the scalability of fine-tuning and pairs it with a prequential evaluation protocol for data-efficient hyperparameter selection. Across Big Bench Hard, NLP task suites, and FLORES translations, IC L+FT consistently matches or outperforms both ICL-Only and FT-Only, often with smaller models and less data. The work also demonstrates robustness to hyperparameters, compatibility with LoRA, and practical guidance for data-scarce scenarios in real-world deployment.

Abstract

When adapting large language models (LLMs) to a specific downstream task, two primary approaches are commonly employed: (1) prompt engineering, often with in-context few-shot learning, leveraging the model's inherent generalization abilities, and (2) fine-tuning on task-specific data, directly optimizing the model's parameters. While prompt-based methods excel in few-shot scenarios, their effectiveness often plateaus as more data becomes available. Conversely, fine-tuning scales well with data but may underperform when training examples are scarce. We investigate a unified approach that bridges these two paradigms by incorporating in-context learning directly into the fine-tuning process. Specifically, we fine-tune the model on task-specific data augmented with in-context examples, mimicking the structure of k-shot prompts. This approach, while requiring per-task fine-tuning, combines the sample efficiency of in-context learning with the performance gains of fine-tuning, leading to a method that consistently matches and often significantly exceeds both these baselines. To perform hyperparameter selection in the low-data regime, we propose to use prequential evaluation, which eliminates the need for expensive cross-validation and leverages all available data for training while simultaneously providing a robust validation signal. We conduct an extensive empirical study to determine which adaptation paradigm - fine-tuning, in-context learning, or our proposed unified approach offers the best predictive performance on a concrete data downstream-tasks.

Figures (22)

• Figure 1: Two representative results from the Big-Bench-Hard suite with between 3 and 150 ground truth examples. We observe the typical signature where ICL+FT performs on-par with ICL-Only when very few examples are available, but improves similarly or better than FT-Only with more training data. Appendix \ref{['sec:bbh-all']} shows the individual results on all 23 BBH tasks.
• Figure 2: Two representative results from the Big-Bench-Hard suite with between 3 and 150 ground truth examples. We observe the typical signature where ICL+FT performs on-par with ICL-Only when very few examples are available, but improves similarly or better than FT-Only with more training data. Appendix \ref{['sec:bbh-all']} shows the individual results on all 23 BBH tasks.
• Figure 3: Average test-set performance across 23 BBH tasks. Left: Performance as a function of the number of ground-truth examples (3 to 150) for a fixed model size. Right: Performance as a function of model size (2B to 27B parameters) for a fixed number of training examples.
• Figure 4: Left: Parity-20 task from Agarwal2024-wpRight: FLoRes English to Kurdish translation task with Gemma-2 27B.
• Figure 5: Left: We revisit the BBH Navigate task with flipped labels (Fig. \ref{['fig:flipped-labels']}), however optionally provide an instruction about the flipped labels in the prompt. Right: Varying the number of in-context examples used during training and testing, averaged over a subset of the BBH tasks. The left most point corresponds to FT-Only .