Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation

TL;DR

The paper conducts a rigorous, model-size–matched comparison of few-shot fine-tuning and in-context learning for task adaptation, using identical OPT models and datasets across 125M–30B parameters. It shows that fine-tuned models can generalize to out-of-domain data as well as, or sometimes better than, in-context learning, challenging the view that FT inherently suffers from poorer OOD generalization. Generalization performance in both approaches exhibits high variance and is strongly influenced by model size, the amount of training/data, and prompt/pattern choices. The study also demonstrates that model-selection strategy and training data scale significantly impact OOD performance, and that PBFT with pattern-based perturbations extends robust gains beyond vanilla FT, with results generalizing to non-OPT architectures like Pythia. Overall, robust task adaptation remains challenging, but fair comparisons reveal that FT and ICL are more similar in practice than previously thought, with clear practical guidance on when to employ each approach.

Abstract

Few-shot fine-tuning and in-context learning are two alternative strategies for task adaptation of pre-trained language models. Recently, in-context learning has gained popularity over fine-tuning due to its simplicity and improved out-of-domain generalization, and because extensive evidence shows that fine-tuned models pick up on spurious correlations. Unfortunately, previous comparisons of the two approaches were done using models of different sizes. This raises the question of whether the observed weaker out-of-domain generalization of fine-tuned models is an inherent property of fine-tuning or a limitation of the experimental setup. In this paper, we compare the generalization of few-shot fine-tuning and in-context learning to challenge datasets, while controlling for the models used, the number of examples, and the number of parameters, ranging from 125M to 30B. Our results show that fine-tuned language models can in fact generalize well out-of-domain. We find that both approaches generalize similarly; they exhibit large variation and depend on properties such as model size and the number of examples, highlighting that robust task adaptation remains a challenge.

Figures (18)

• Figure 1: In-domain (RTE) and out-of-domain performance (HANS) for in-context learning (ICL) and fine-tuning (FT) with OPT models of various sizes. We fine-tune models using pattern-based fine-tuning. We report results using 10 different data seeds. When using 16 samples, ICL's performance with a 30B model is comparable to that of FT with smaller models (6.7B) and for most model sizes, FT outperforms ICL (see \ref{['tab:appendix-statistical-tests-in-domain-ood-rte']} for significance tests). $\boldsymbol{-}$ in the x- and y-axes indicates majority class accuracy.
• Figure 2: ICL and FT results for OPT models of various sizes. For each approach, we use 16 examples and perform model selection according to OOD performance. We plot 10 runs per model size which differ only in the data seed. $\boldsymbol{-}$ in the x- and y-axis indicates majority class accuracy.
• Figure 3: Comparing model selection strategies in FT. The first and second rows show results for MNLI and RTE respectively. We train on 16 examples and plot results for 10 runs for each model size. $\boldsymbol{-}$ in the x- and y-axes indicates majority class accuracy.
• Figure 4: Exploring the effect of increasing training examples on FT. The first and second rows show results for MNLI and RTE respectively. We plot results for 10 runs for each model size and perform model selection according to out-of-domain performance. $\boldsymbol{-}$ in the x- and y-axes indicates majority class accuracy.
• Figure 5: Estimating OOD performance using less data. We compare OOD performance estimated using all vs. 50 examples when fine-tuning OPT 13B on RTE. Each color corresponds to a run with a different data seed.