"Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification

TL;DR

The paper tackles the problem of unreliable faithfulness among input salience methods for text classification by proposing a ground-truth–driven protocol that uses partially synthetic data to instantiate lexical shortcuts as ground-truth token importance. It defines two metrics, $p@k$ and $rank$, and details a protocol to inject shortcuts, train models, verify reliance on shortcuts, and evaluate salience methods (Gradient, Gradient times Input, Integrated Gradients, and LIME) across BERT and LSTM on multiple datasets. Across three shortcut types (single-token, token-in-context, and ordered-pair), the authors show that method effectiveness is highly task- and model-dependent, with Grad-L2 performing strongly for BERT while some IG configurations underperform, especially when baselines or perturbation strategies are mis-specified. The work provides practical guidance for debugging NLP models and detecting shortcuts, and offers a reusable protocol that can be applied to new tasks and architectures to identify the most faithful and useful salience method for a given setting.

Abstract

Feature attribution a.k.a. input salience methods which assign an importance score to a feature are abundant but may produce surprisingly different results for the same model on the same input. While differences are expected if disparate definitions of importance are assumed, most methods claim to provide faithful attributions and point at the features most relevant for a model's prediction. Existing work on faithfulness evaluation is not conclusive and does not provide a clear answer as to how different methods are to be compared. Focusing on text classification and the model debugging scenario, our main contribution is a protocol for faithfulness evaluation that makes use of partially synthetic data to obtain ground truth for feature importance ranking. Following the protocol, we do an in-depth analysis of four standard salience method classes on a range of datasets and shortcuts for BERT and LSTM models and demonstrate that some of the most popular method configurations provide poor results even for simplest shortcuts. We recommend following the protocol for each new task and model combination to find the best method for identifying shortcuts.

Figures (3)

• Figure 1: Salience maps produced by four common methods on a sentiment classification example (SST2) for a BERT model. The same token (eastwood) is assigned the highest (Grad-L2), the lowest (GxI, LIME) and a mid-range (IG) importance score (color intensity indicates salience; blue and purple stand for positive, red stands for negative weights). A developer investigating a hypothesis about specific named entities being associated with the label would probably be unsure as to whether the example provides support for or against the hypothesis.
• Figure 2: The proposed protocol to obtain ground truth importance rankings.
• Figure 3: Illustration of how the ordered-pair shortcut is introduced into a balanced binary sentiment dataset and how it is verified that the shortcut is learned by the model. The model trained on the mixed data (A) is still largely a black box, but since its performance on the synthetic test set is 100% (contrasted with chance accuracy of model B which is similar but is trained on the original data only), we know it uses the injected shortcut (highlighted text).