Attention Interpretability Across NLP Tasks

TL;DR

The paper tackles conflicting claims about attention interpretability in NLP by conducting a broad empirical analysis across text classification, natural language inference, and neural machine translation, using multiple attention variants and self-attention models. It shows that attention can be interpretable and correlate with feature importance in tasks where it is essential to prediction, particularly in pairwise and generation tasks, while often behaving as a gating mechanism in single-sequence tasks. Through systematic weight perturbations, permutation studies, and manual interpretability evaluations, the work reconciles earlier contradictory findings and delineates the conditions under which attention serves as a reliable explanation. Overall, it clarifies when attention weights reflect underlying reasoning and when they function primarily as gating signals within the network.

Abstract

The attention layer in a neural network model provides insights into the model's reasoning behind its prediction, which are usually criticized for being opaque. Recently, seemingly contradictory viewpoints have emerged about the interpretability of attention weights (Jain & Wallace, 2019; Vig & Belinkov, 2019). Amid such confusion arises the need to understand attention mechanism more systematically. In this work, we attempt to fill this gap by giving a comprehensive explanation which justifies both kinds of observations (i.e., when is attention interpretable and when it is not). Through a series of experiments on diverse NLP tasks, we validate our observations and reinforce our claim of interpretability of attention through manual evaluation.

Figures (5)

• Figure 1: Comparison of performance with and without neural attention on text classification (IMDB), Natural Language Inference tasks (SNLI) and Neural Machine Translation (News Commentary). Here, $\alpha$ and $c$ denote attention weights and context vector respectively. The results show that attention does not substantially effect performance on text classification. However, the same does not hold for other tasks.
• Figure 2: Relationship between maximum attention weight and median change in output on permuting attention weights. For single sequence tasks, $\blacksquare$, $\blacksquare$ indicate negative and positive class. For MultiNLI, $\blacksquare$, $\blacksquare$, $\blacksquare$ denotes contradiction, entailment and neutral respectively. The results reinforce the claim that altering attention weights in single sequence tasks does not have much effect on performance while the same does not hold with other tasks. Refer to §\ref{['sec:results_alt_atten']} for details.
• Figure 3: Analysis of correlation between attention weights and feature importance measure. We report relationship between difference in zeroed attention weights and corresponding change in $JS$ divergence for different tasks. Please refer to §\ref{['sec:results_feature_imp']} for more details.
• Figure 4: Performance comparison with permuting attention weights for different layers in self-attention based models. The results are reported on a representative instance of single sequence, pair sequence and generation tasks. The dotted lines denote the base performance on the task. Refer to §\ref{['sec:results_self_attention']} for details.
• Figure 5: Manual evaluation of interpretability of attention weights on single and pair sequence tasks. Although with original weights the attention does remain interpretable on both tasks but in the case of single sequence tasks making it meaningless does not change the prediction substantially. However, the same does not hold with pair sequence tasks.

Theorems & Definitions (1)

• Proposition 4.1