Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals

TL;DR

This work introduces Counterfactual Attentiveness Test (CAT) to quantify whether full-input NLP models attend to partial inputs by replacing a portion of the input with a counterfactual from another example and observing if the model’s prediction changes. CAT is applied across ten datasets spanning four tasks (NLI, PD, RC, VLR) and evaluating both supervised finetuning and in-context learning models, revealing that attentiveness is highly data- and task-dependent; in many cases, full-input models do not rely on known correlations, yet targeted counterfactual augmentation can improve attentiveness. The study also shows that more demonstrations in in-context learning can reduce attentiveness for some models (e.g., GPT-3) even as standard accuracy improves, highlighting memorization as a potential factor. CAT augmentation demonstrates practical gains in attentiveness with minimal impact on standard performance, offering a simple tool to diagnose and strengthen model robustness beyond traditional correlation-based analyses.

Abstract

The inevitable appearance of spurious correlations in training datasets hurts the generalization of NLP models on unseen data. Previous work has found that datasets with paired inputs are prone to correlations between a specific part of the input (e.g., the hypothesis in NLI) and the label; consequently, models trained only on those outperform chance. Are these correlations picked up by models trained on the full input data? To address this question, we propose a new evaluation method, Counterfactual Attentiveness Test (CAT). CAT uses counterfactuals by replacing part of the input with its counterpart from a different example (subject to some restrictions), expecting an attentive model to change its prediction. Using CAT, we systematically investigate established supervised and in-context learning models on ten datasets spanning four tasks: natural language inference, reading comprehension, paraphrase detection, and visual & language reasoning. CAT reveals that reliance on such correlations is mainly data-dependent. Surprisingly, we find that GPT3 becomes less attentive with an increased number of demonstrations, while its accuracy on the test data improves. Our results demonstrate that augmenting training or demonstration data with counterfactuals is effective in improving models' attentiveness. We show that models' attentiveness measured by CAT reveals different conclusions from solely measuring correlations in data.

Figures (2)

• Figure 1: We are interested in quantifying model's attentiveness to part of the input. We propose CAT, an evaluation that replaces the premise $P$ with a counterfactual $P'$. A change in behavior to the counterfactual input indicates the model is attentive to the premise, otherwise, the model relies solely on the hypothesis to make a prediction.
• Figure 2: Attentiveness as a function of partial input correlations for the supervised models. Higher attentiveness values ($y$ axis) indicate better attentiveness to counterfactual inputs. Higher values on the partial input correlations indicate correlations between a part of the input and the label. It is computed as the score on the partial input baseline, subtracting the majority score.