Attention is not not Explanation

TL;DR

This paper revisits the claim that attention is not explanation by Jain & Wallace, arguing that such conclusions hinge on experimental design and definitions. It introduces four model-driven tests—uniform attention baselines, seed-based variance analysis, diagnostic non-contextual MLPs, and model-consistent adversarial training—to assess attention's usefulness for explanation in LSTM-based architectures. Across multiple NLP datasets, the authors show that uniform attention can match performance in some tasks, variance exists across seeds, and adversarial attention can be constructed but often fails to transfer to meaningful, non-contextual explanations, suggesting that attention can be informative under careful evaluation. Finally, the work clarifies interpretability versus explainability definitions, offers a rigorous evaluation framework, and highlights dataset-dependent conditions under which attention provides faithful or plausible explanations, with implications for debugging and model design.

Abstract

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

Figures (6)

• Figure 1: Schematic diagram of a classification LSTM model with attention, including the components manipulated or replaced in the experiments performed in jain2019attention and in this work (by section).
• Figure 2: Attention maps for an IMDb instance (all predicted as positive with score $> 0.998$), showing that in practice it is difficult to learn a distant adversary which is consistent on all instances in the training set.
• Figure 3: Densities of maximum JS divergences (x-axis) as a function of the max attention (y-axis) in each instance between the base distributions and: (a-d) models initialized on different random seeds; (e-f) models from a per-instance adversarial setup (replication of Figure 8a, 8c resp. in jain2019attention). In each max-attention bin, top (blue) is the negative-label instances, bottom (red) positive-label instances.
• Figure 4: Diagram of the setup in § \ref{['ssec:guide']} (except Trained MLP, which learns weight parameters).
• Figure 5: Averaged per-instance test set JSD and TVD from base model for each model variant. JSD is bounded at $\sim 0.693$. $\blacktriangle$: random seed; $\blacksquare$: uniform weights; dotted line: our adversarial setup as $\lambda$ is varied; $\boldsymbol{+}$: adversarial setup from jain2019attention.