Mechanisms are Transferable: Data-Efficient Low-Resource Adaptation via Circuit-Targeted Supervised Fine-Tuning

TL;DR

Circuit-Targeted Supervised Fine-Tuning (CT-SFT) is proposed, a counterfactual-free adaptation of CD-T that uses a label-balanced mean baseline and task-directional relevance scoring to identify a sparse set of task-relevant attention heads in a proxy-language checkpoint, then transfer learns to a target language by updating only those heads via head-level gradient masking.

Abstract

Adapting LLMs to low-resource languages is difficult: labeled data is scarce, full-model fine-tuning is unstable, and continued cross-lingual tuning can cause catastrophic forgetting. We propose Circuit-Targeted Supervised Fine-Tuning (CT-SFT): a counterfactual-free adaptation of CD-T (Contextual Decomposition Transformer) that uses a label-balanced mean baseline and task-directional relevance scoring to identify a sparse set of task-relevant attention heads in a proxy-language checkpoint, then transfer learns to a target language by updating only those heads (plus LayerNorm) via head-level gradient masking. Across NusaX-Senti and XNLI, CT-SFT improves cross-lingual accuracy over continued full fine-tuning while updating only a small subset of model parameters. We find an editing-preserving trade-off: harder transfers favor editing circuit heads, while easier transfers often favor near-zero (i.e., low-relevance heads) updates, preserving the source mechanism. CT-SFT also substantially reduces catastrophic forgetting, preserving proxy/source-language competence during transfer.

Figures (8)

• Figure 1: CT-SFT overview. Our method has two phases. Mechanism Discovery (left): starting from a base model, we perform competence tuning on a higher-resource proxy language to obtain a task-competent checkpoint $\theta_1$. From a discovery pool, we form (i) a correctly-predicted discovery set $\mathcal{X}$ under $\theta_1$ and (ii) a label-balanced statistical baseline $\mu$ for CD-T. We then run a modified CD-T procedure to score components and select a task-relevant circuit $\mathcal{C}$ (a subset of attention heads). We optionally compute circuit-faithfulness diagnostics on a validation set. Mechanism-Guided Tuning (middle): for a low-resource target language, we fine-tune only the parameters corresponding to heads in $\mathcal{C}$ (and LayerNorm), using a held-out tuning pool, producing the CT-SFT checkpoint $\theta_{\text{ct-sft}}$. Evaluation (right): we evaluate $\theta_{\text{ct-sft}}$ on the test set and report task metrics.
• Figure 2: Indonesian $\rightarrow$ Indonesian: test accuracy versus training sample size under different head-selection strategies, shown for circuit expansion depths (max_depth $=0,1,2$).
• Figure 3: Accuracy-based faithfulness across circuit expansion depths comparing magnitude-based (norm) scoring and our task-direction projection scoring.
• Figure 4: Cross-lingual NusaX transfer (Indonesian $\rightarrow$ target): test accuracy vs. target-language tuning size $n$, shown per target language. Each panel contains three subplots corresponding to circuit expansion depth (max_depth$=0,1,2$) under the surgical setting, comparing Circuit CT-SFT against control head selections. Dashed horizontal lines indicate the competence-tuning baseline (Indonesian-tuned checkpoint evaluated directly on the target language, with no Stage-2 tuning).
• Figure 5: XNLI cross-lingual transfer results (surgical setting; 250-sample English competence tuning): test accuracy vs. tuning set size $n$ across circuit depths. We compare circuit-targeted updates to near-zero head updates for multiple target languages.