Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transformer Interpretability Beyond Attention Visualization

Hila Chefer, Shir Gur, Lior Wolf

TL;DR

This paper presents a Deep Taylor Decomposition–based relevancy propagation method for Transformer networks that preserves total relevancy across layers, addressing skip connections and nonlinearities. By integrating gradients with attention maps and normalizing binary operations, it produces class-specific explanations that propagate from output back to input tokens. The approach yields state-of-the-art results on both vision (ViT) and NLP (BERT) explainability benchmarks, outperforming rollout, Grad-CAM, and LRP baselines on perturbation, segmentation, and rationales tasks. The work offers a practical, mechanistic, and faithful framework for interpreting Transformer decisions with broad applicability in vision and language domains.

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

Transformer Interpretability Beyond Attention Visualization

TL;DR

This paper presents a Deep Taylor Decomposition–based relevancy propagation method for Transformer networks that preserves total relevancy across layers, addressing skip connections and nonlinearities. By integrating gradients with attention maps and normalizing binary operations, it produces class-specific explanations that propagate from output back to input tokens. The approach yields state-of-the-art results on both vision (ViT) and NLP (BERT) explainability benchmarks, outperforming rollout, Grad-CAM, and LRP baselines on perturbation, segmentation, and rationales tasks. The work offers a practical, mechanistic, and faithful framework for interpreting Transformer decisions with broad applicability in vision and language domains.

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

Paper Structure

This paper contains 22 sections, 4 theorems, 24 equations, 34 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Explainability in computer vision
  4. Explainability for Transformers
  5. Method
  6. Relevance and gradients
  7. Non parametric relevance propagation:
  8. Relevance and gradient diffusion
  9. Obtaining the image relevance map
  10. Experiments
  11. Results
  12. Conclusions
  13. Details of the various Baselines
  14. GradCAM
  15. raw-attention
  16. ...and 7 more sections

Key Result

Lemma 1

Given two tensors $u$ and $v$, consider the relevances that are computed according to Eq. eq:uvr. Then, (i) if layer $L^{(n)}$ adds the two tensors, i.e., $L^{(n)}(u,v) = u+v$ then the conservation rule of Eq. eq:binarycr is maintained. (ii) if the layer performs matrix multiplication $L^{(n)}(u,v)

Figures (34)

  • Figure 1: Illustration of our method. Gradients and relevancies are propagated through the network, and integrated to produce the final relevancy maps, as described in Eq. \ref{['eq:modified_att']}, \ref{['eq:modified_rollout']}.
  • Figure 1: Multiple-class visualization. For each input image, we visualize two different classes. As can be seen, only our method and GradCAM produce class-specific visualisations, where our method has fewer artifacts, and captures the objects more completely.
  • Figure 2: Sample results. As can be seen, our method produces more accurate visualizations.
  • Figure 2: Multiple-class visualization. For each input image, we visualize two different classes. As can be seen, only our method and GradCAM produce class-specific visualisations, where our method has fewer artifacts, and captures the objects more completely.
  • Figure 3: Sample images from ImageNet val-set.
  • ...and 29 more figures

Theorems & Definitions (8)

  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Lemma 4
  • proof