Assessing Robustness to Spurious Correlations in Post-Training Language Models

TL;DR

The paper tackles robustness to spurious correlations in post-training language-model alignment by building a synthetic benchmark with FA and DN manipulations across docQA, math, and constrained-instruction tasks. It systematically compares SFT, Direct Preference Optimization ($\text{DPO}$), and Kahneman-Tversky Optimization ($\text{KTO}$) across spuriousness levels of $10\%$ vs $90\%$ and analyzes results on open LLMs. Key findings show that $\text{DPO}$/$\text{KTO}$ exhibit math-task robustness, while $\text{SFT}$ often preserves performance in docQA; instructional tasks tend to falter under high spuriousness. The study underscores that no single post-training strategy universally outperforms others; task type and spurious-profile should guide method choice, with implications for real-world deployment and potential hybrids or data-denoising approaches.

Abstract

Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world training data often exhibits spurious correlations -- arising from biases, dataset artifacts, or other "shortcut" features -- that can compromise a model's performance or generalization. In this paper, we systematically evaluate three post-training algorithms -- Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and KTO (Kahneman-Tversky Optimization) -- across a diverse set of synthetic tasks and spuriousness conditions. Our tasks span mathematical reasoning, constrained instruction-following, and document-grounded question answering. We vary the degree of spurious correlation (10% vs. 90%) and investigate two forms of artifacts: "Feature Ambiguity" and "Distributional Narrowness." Our results show that the models often but not always degrade under higher spuriousness. The preference-based methods (DPO/KTO) can demonstrate relative robustness in mathematical reasoning tasks. By contrast, SFT maintains stronger performance in complex, context-intensive tasks. These findings highlight that no single post-training strategy universally outperforms in all scenarios; the best choice depends on the type of target task and the nature of spurious correlations.

Figures (3)

• Figure 1: docQA tasks: arrows connect each model's accuracy at 10% to its accuracy at 90% spuriousness. Marker style denotes the model size, color denotes the training method. Omission shows a large drop in accuracy from 10% to 90% spuriousness, and SFT outperforms preference methods in the other docQA tasks.
• Figure 2: math tasks: each point is a (model, method) at 10% or 90% spuriousness, with arrows illustrating accuracy shifts between the two. DPO and KTO outperform SFT for these tasks, with similar accuracy or slight rise in accuracy from increasing spuriousness.
• Figure 3: instruction tasks: performance for 10% vs. 90% spurious data. Models and methods are distinguished by markers and colors, respectively. Most settings drop in accuracy for the "tiny constraints" task when spuriousness increases, and overall accuracy is very low for the other two tasks.