Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance

TL;DR

This work reveals a surprising failure mode of instruction finetuning: under knowledge conflicts, models initially increase reliance on user-provided context but progressively shift back to parametric memory, even as standard task performance improves. The authors develop a theoretical framework using a simplified one-layer transformer to distinguish context-critical versus non-context-critical datapoints and show how gradient signals shift over training to reduce context reliance. Extensive experiments across model families (Llama, Pythia, Mistral) and instruction-tuning datasets (TULU, UltraChat, Alpaca) confirm the context-parametric inversion and show that non-context-critical datapoints largely drive the later decline; targeted data curation can mitigate it but only partially. The work highlights a fundamental limitation of current instruction finetuning pipelines and provides empirical and theoretical insights toward strategies to better align context usage with user instructions in real-world deployments.

Abstract

A standard practice when using large language models is for users to supplement their instruction with an input context containing new information for the model to process. However, models struggle to reliably follow the input context, especially when it conflicts with their parametric knowledge from pretraining. In-principle, one would expect models to adapt to the user context better after instruction finetuning, particularly when handling knowledge conflicts. However, we observe a surprising failure mode: during instruction tuning, the context reliance under knowledge conflicts initially increases as expected, but then gradually decreases as instruction finetuning progresses. This happens while the performance on standard benchmarks keeps on increasing far after this drop. We call this phenomenon context-parametric inversion and observe it across multiple general purpose instruction tuning datasets such as TULU, Alpaca and Ultrachat, across different model families like Llama, Mistral, and Pythia. We perform various controlled studies and theoretical analysis to show that context-parametric inversion occurs due to examples in the instruction finetuning data where the input context provides information that aligns with model's parametric knowledge. Our analysis suggests some natural mitigation strategies with limited but insightful gains, and serves as a useful starting point in addressing this deficiency in instruction finetuning.

Figures (15)

• Figure 1: (a) Context-Parametric Inversion In the presence of conflicts between parametric and context information, a model's reliance on context first increases and then decreases during the process of instruction finetuning. (b) Instruction tuning data includes both context-critical examples, where the context is essential, and non-context-critical examples, where responses can be generated using either the context or the model’s parametric knowledge. It is this latter group of non-context-critical examples that leads to a decline in context reliance (§ \ref{['sec:data_categoreis']}).
• Figure 2: Accuracy on Different Knowledge Conflict Datasets We track how the model's context reliance (over parametric memory) evolves during instruction fine-tuning, particularly under knowledge conflicts. Counterfactual (blue) and parametric (orange) accuracy on (a) CF Quotes, (b) Biographics, and (c) World Facts versus average performance on standard benchmarks (GSM8k, MMLU, ARC, SQuAD).
• Figure 3: Not Overfitting (a) Peak performance on CF_Quotes occurs well before the end of one epoch. (b) Attention score of LLama7B over the context for the CF_World_Facts eval set averaged over all the layers. Consistent with our theory (§ \ref{['sec:theory']}), the attention to context rises and falls. We do not make any causal claims from this observation about the attention dynamic in deep networks.
• Figure 4: Filtering Harmful Examples (a) Controlling for fact overlap between train-test sets, we still observe a drop in context reliance. (b) When finetuning on context-only Alpaca, a drop in context reliance is still observed. However, on a context-critical subset of Alpaca, there is no drop. (c) The drop in context reliance happens when finetuning on context-based QA datasets like SQuAD.
• Figure 5: Mitigation Strategies (a, b) Counterfactual data augmentation mitigates drop in context reliance on some tasks similar to the augmented data, but doesn't generalize across all tasks (§ \ref{['sec:augmentation']}). (c) Only updating the query and key matrices can give potential gains but at the cost of standard benchmark performance (§ \ref{['sec:qk']})

Theorems & Definitions (29)

• Proposition 1
• Proposition 2: More Attention to Subject with S Points
• Proposition 3: Fact Memorization
• Theorem 1: Test-Time Dynamic
• Definition 1: Memorization
• Definition 2: C Datapoints
• Definition 3: C+S Datapoints
• Proposition 1
• proof
• Lemma 1