Think Twice: Measuring the Efficiency of Eliminating Prediction Shortcuts of Question Answering Models

TL;DR

This work introduces a bootstrapped framework to quantify how QA models rely on spurious, bias-driven features, arguing that traditional OOD evaluation alone can misrepresent robustness when biases are shared across datasets. By evaluating predictions on bias-based data splits for seven features and applying debiasing methods (ReSam, LMix, CReg), the study reveals that reductions in Prediction bias do not consistently translate to improved OOD performance, and that biases can transfer across QA datasets. The authors show that model pre-training scale generally reduces reliance on biases, while debiasing methods vary in effectiveness across biases and can slow training. They also demonstrate that models trained on one QA dataset can exhibit similar bias reliance when evaluated on others, underscoring the need for bias-aware robustness reporting. Overall, the paper advocates for complementary bias-focused analyses to better gauge LMs’ robustness beyond OOD metrics, guiding more reliable deployment of QA systems.

Abstract

While the Large Language Models (LLMs) dominate a majority of language understanding tasks, previous work shows that some of these results are supported by modelling spurious correlations of training datasets. Authors commonly assess model robustness by evaluating their models on out-of-distribution (OOD) datasets of the same task, but these datasets might share the bias of the training dataset. We propose a simple method for measuring a scale of models' reliance on any identified spurious feature and assess the robustness towards a large set of known and newly found prediction biases for various pre-trained models and debiasing methods in Question Answering (QA). We find that while existing debiasing methods can mitigate reliance on a chosen spurious feature, the OOD performance gains of these methods can not be explained by mitigated reliance on biased features, suggesting that biases are shared among different QA datasets. Finally, we evidence this to be the case by measuring that the performance of models trained on different QA datasets relies comparably on the same bias features. We hope these results will motivate future work to refine the reports of LMs' robustness to a level of adversarial samples addressing specific spurious features.

Figures (9)

• Figure 1: We quantify model reliance on a spurious feature using bootstrapped evaluation on segments of data separated by exploiting chosen bias (left) and subsequently, by measuring the difference in model's performance over these two groups (right), that we refer to as Prediction bias (§\ref{['sec:measusing']}).
• Figure 2: Prediction bias per pre-trained model. The worse-performing split performance (lower bars) and Prediction bias (upper bars, sorted by group average) of QA models trained from different pre-trained LLMs, trained and evaluated on SQuAD for Exact match. Per-group bootstrapping of 100 repeats with 800 samples.
• Figure 3: OOD performance per pre-trained model. Comparison of F1-score of different models fine-tuned on SQuAD and evaluated on listed OOD datasets.
• Figure 4: Prediction bias per dataset. The worse-performing split performance (lower bars) and Prediction bias (upper bars) of RoBERTa-Large trained on different QA datasets, evaluated on a validation split of SQuAD for Exact match. All evaluation splits are identical, identified as maximal for the SQuAD-trained model (Appx. \ref{['apx:heuristics']}).
• Figure 5: Prediction bias per debiasing methods. The worse-performing split performance (lower bars) and Prediction bias (upper bars) of BERT-Base trained using selected debiasing methods, evaluated for Exact match on validation SQuAD. Per-group evaluations were measured using bootstrapping of 100 repeats with 800 samples.