In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax

TL;DR

Problem: determine whether in-context learning (ICL) in large language models captures underlying syntactic structure or relies on superficial cues under distribution shifts. Approach: a suite of syntactic transformation tasks and a syntactic NLI evaluation (Hans), tested across GPT, PaLM, and Llama 2 families with in-context guidance and chain-of-thought prompting. Findings: substantial inter-model variance not explained by size; code pre-training improves out-of-distribution generalization and reasoning faithfulness, while chain-of-thought prompts can improve in-distribution results at the expense of OOD performance; RLHF may Harm generalization. Significance: highlights limits of ICL for robust language understanding and suggests that code-based pre-training and careful OOD evaluation are crucial for reliable generalization in LLMs.

Abstract

In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates. Do models guided via ICL infer the underlying structure of the task defined by the context, or do they rely on superficial heuristics that only generalize to identically distributed examples? We address this question using transformations tasks and an NLI task that assess sensitivity to syntax - a requirement for robust language understanding. We further investigate whether out-of-distribution generalization can be improved via chain-of-thought prompting, where the model is provided with a sequence of intermediate computation steps that illustrate how the task ought to be performed. In experiments with models from the GPT, PaLM, and Llama 2 families, we find large variance across LMs. The variance is explained more by the composition of the pre-training corpus and supervision methods than by model size; in particular, models pre-trained on code generalize better, and benefit more from chain-of-thought prompting.

Figures (8)

• Figure 1: The syntactic transformations paradigm. We prompt language models with labeled examples $(x,y)_\text{train}$ that can be explained using either robust syntactic/hierarchical features or spurious positional/linear features. We also include the input from a test example $x_\text{test}$. We ensure the models have learned the task by evaluating on in-distribution examples $(x,y)_\text{test-id}$. Then, we observe whether models generalize syntactically or linearly on out-of-distribution examples $(x,y)_\text{test-ood}$.
• Figure 2: The prompt formats we use for question formation. We give the model up to 8 exemplars, followed by a test example. The model must generate the chain-of-thought reasoning and the answer. We highlight the answers with a blue background. In the code prompt, we put code (except comments) in blue text. We construct similar prompts for tense reinflection; see Appendix \ref{['app:tense_reinflection_prompts']}.
• Figure 3: Main auxiliary accuracy on question formation and verb accuracy on tense reinflection. In-distribution accuracies reveal whether the models have learned the task, and out-of-distribution accuracies reveal whether models generalize robustly. Unfilled shapes (GPT-3, Llama 2) were trained on less than 5% code. We interpret the dashed line as a ceiling on OOD accuracy given ID accuracy.
• Figure 4: Reasoning accuracies and faithfulness scores for question formation (left) and tense reinflection (right) using the Code CoT prompt. Reasoning accuracies and faithfulness are highest for code-davinci-002, GPT-4, (Flan-)PaLM, and CodeLlama.
• Figure 5: The prompt formats we use for tense reinflection. We give the model up to 8 exemplars, followed by a test example. The model must generate the chain-of-thought reasoning and the answer. We highlight the answers with a blue background. In the code prompt, we put code (except comments) in blue text.