On the generalization of language models from in-context learning and finetuning: a controlled study

TL;DR

A method is proposed to enable improved generalization from fine-tuning: adding in-context reasoning traces to finetuning data, and it is shown that this method improves generalization across various splits of the authors' datasets and other benchmarks.

Abstract

Large language models exhibit exciting capabilities, yet can show surprisingly narrow generalization from finetuning. E.g. they can fail to generalize to simple reversals of relations they are trained on, or fail to make simple logical deductions based on trained information. These failures to generalize factual information from fine-tuning can significantly hinder the reasoning capabilities of these models. On the other hand, language models' in-context learning (ICL) shows different inductive biases and deductive reasoning capabilities. Here, we explore these differences in generalization and deductive reasoning between in-context- and fine-tuning-based learning. To do so, we constructed several novel datasets to evaluate and improve models' abilities to make generalizations over factual information from novel data. These datasets are designed to create clean tests of generalization, by isolating the knowledge in the dataset from that in pretraining. We expose pretrained large models to controlled subsets of the information in these datasets -- either through ICL or fine-tuning -- and evaluate their performance on test sets that require various types of generalization. We find overall that in data-matched settings, ICL can generalize several types of inferences more flexibly than fine-tuning (though we also find some qualifications of prior findings, such as cases when fine-tuning can generalize to reversals embedded in a larger structure of knowledge). We build on these findings to propose a method to enable improved generalization from fine-tuning: adding in-context reasoning traces to finetuning data. We show that this method improves generalization across various splits of our datasets and other benchmarks. Our results have implications for understanding the generalization afforded by different modes of learning in language models, and practically improving their performance.

Figures (13)

• Figure 1: A semantic structure with a hierarchy of properties and relations. (Reproduced with permission from rogers2008precis.)
• Figure 2: Reversal curse paper results.
• Figure 3: On our simple reversal (left) and syllogism (right) datasets, in-context learning outperforms finetuning. Moreover, augmenting the fine-tuning dataset produces strong improvements in model performance. Pretrained models perform near chance, showing that the datasets are not guessable based on superficial features.
• Figure 4: On the more richly structured semantic dataset, in-context learning still moderately outperforms finetuning. Furthermore, augmentation continues to show benefit for generalization from finetuning --- even in rephrased questions about trained facts that do not involve reversal (top left). However some generalization splits, such as the category-level holdout, remain very challenging. (Error bars are standard errors computed over task subsets featuring different types of the inferences in question, e.g. reversals of property relations vs. reversals of category inclusion relations.)
• Figure 5: Sentence splitting analysis on the semantic structure benchmark. These plots show the finetuning performance when the documents in the training dataset are split at sentence-level. We can observe that except for augmented variants of the dataset, sentence splitting consistently improves performance.