Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions

TL;DR

The paper tackles the opacity of modern NLP models by comparing gradient-based saliency maps with influence functions as explanations for predictions. Influence functions connect test predictions to influential training examples via $-\frac{d\mathcal{L}_{\hat{y}}}{d\epsilon_i}$, and are evaluated on BERT for sentiment analysis and natural language inference, with a focus on artifact detection using the HANS diagnostic set. The authors demonstrate that influence functions can be reliably applied to deep transformers for sentiment analysis and can reveal data artifacts in NLI, though they show limited alignment with saliency methods in the NLI domain. A quantitative artifact-measure based on influence-artifact distributions (involving a quadratic coefficient) is introduced to surface and quantify artifacts like lexical overlap and negation, highlighting their interaction and impact on predictions. Overall, influence functions offer a valuable, complementary lens for model interpretation and data auditing in NLP, particularly for complex reasoning tasks and artifact discovery.

Abstract

Modern deep learning models for NLP are notoriously opaque. This has motivated the development of methods for interpreting such models, e.g., via gradient-based saliency maps or the visualization of attention weights. Such approaches aim to provide explanations for a particular model prediction by highlighting important words in the corresponding input text. While this might be useful for tasks where decisions are explicitly influenced by individual tokens in the input, we suspect that such highlighting is not suitable for tasks where model decisions should be driven by more complex reasoning. In this work, we investigate the use of influence functions for NLP, providing an alternative approach to interpreting neural text classifiers. Influence functions explain the decisions of a model by identifying influential training examples. Despite the promise of this approach, influence functions have not yet been extensively evaluated in the context of NLP, a gap addressed by this work. We conduct a comparison between influence functions and common word-saliency methods on representative tasks. As suspected, we find that influence functions are particularly useful for natural language inference, a task in which 'saliency maps' may not have clear interpretation. Furthermore, we develop a new quantitative measure based on influence functions that can reveal artifacts in training data.

Figures (5)

• Figure 1: A sentiment analysis example interpreted by gradient-based saliency maps (left) and influence functions (right). Note that this example is classified incorrectly by the model. Positive saliency tokens and highly influential examples may suggest why the model makes the wrong decision; tokens and examples with negative saliency or influence scores may decrease the model's confidence in making that decision.
• Figure 2: Average influence score of top sentiment analysis training examples that contain a token in test example with most positive, most negative, or median saliency. Error bars depict standard errors.
• Figure 3: Average influence score of top NLI training examples that contain a token in test example with most positive, most negative, or median saliency. Standard error is shown in error bars.
• Figure 4: Influence-artifact distribution for different test examples.
• Figure 5: Influence-artifact distribution for an original and negated HANS example. (P: The lawyers saw the professor behind the bankers. H: The lawyers saw / did not see the professor.)