A Unified Causal View of Instruction Tuning

TL;DR

The paper targets spurious correlations in instruction tuning by introducing a meta-Structural Causal Model (meta-SCM) that unifies multiple NLP tasks under a single causal structure. It proves a Uniform Identifiability Condition (UIC) to guarantee identifiability of latent factors and presents Structural Instruction Tuning (SIT), which learns task-specific causal factor selections and causal pathways from factors to targets. The method shows improved zero-shot and cross-task generalization on unseen datasets and tasks, with ablations indicating the necessity of UIC-guided regularization and distinct task-factor learning. This causal-inference framework enhances robustness and transferability of instruction-tuned models, and provides a foundation for extending causality into multitask NLP and beyond, including potential integrations with larger language models.

Abstract

Instruction tuning on a mixture of tasks has improved zero-shot capabilities in natural language processing (NLP). Nevertheless, existing methods often learn features that exhibit correlations between instruction-formatted samples and target labels, rather than causal relationships. Termed as ``spurious correlation'' in statistics, such a correlation may change drastically in a new task, making the effect from the learned features to be misleading. To this end, we develop a meta Structural Causal Model (meta-SCM) to integrate different NLP tasks under a single causal structure of the data. Specifically, the meta-SCM introduces multiple latent factors that represent properties of source context, only some of which causally influence the target labels for a specific task. The key idea is to learn task-required causal factors and only use those to make predictions for a given task. Theoretically, we prove the causal factor can be identified without mixing information from others. Guided by the identifiability, we propose a Structural Instruction Tuning (SIT) method to learn the task-required causal representations that can mimic the causal factors for each task. The utility of our approach is verified by improvements of zero-shot ability on a range of unseen datasets and tasks.

Figures (5)

• Figure 1: Left: The causal graph induced by the meta Structural Causal Model (meta-SCM) for integrating different NLP tasks. White nodes denote observed variables and grey nodes denote unobserved variables. Dashed lines denote edges that may be absent, while solid lines denote invariant processes. Details can be found in Section \ref{['sec:uscm']}. Right: The model overview of Structural Instruction Tuning (SIT), aiming at learning the representations for task-required causal factors. Task information based on prompts guides the causal factor selection. Details can be found in Section \ref{['sec:sit']}.
• Figure 2: Few-shot results.
• Figure 3: Identifiable latent factors.
• Figure 4: Unidentifiable latent factors.
• Figure 5: Overview of the proof. Each step focuses on the element marked in black. In Step 1, we demonstrate that the condition stated in Proposition \ref{['prop:ness']} is a necessary condition for determining SCM identifiability. In Step 2, we establish the equivalence between the conditions in Proposition \ref{['prop:ness']} and Theorem \ref{['theorem:UIM']}, thereby showing that both conditions are necessary and sufficient. Finally, in Step 3, we present the matrix form representation of the condition in Proposition \ref{['prop:ness']}.

Theorems & Definitions (12)

• Definition 3.1: Identifiability
• Theorem 3.2
• Theorem 3.3
• Theorem A.1
• Lemma A.1
• Lemma A.2
• Lemma A.3
• Lemma A.4
• Theorem A.4
• Proposition A.5