Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When LRP Diverges from Leave-One-Out in Transformers

Weiqiu You, Siqi Zeng, Yao-Hung Hubert Tsai, Makoto Yamada, Han Zhao

TL;DR

This paper analyzes why Layer-Wise Relevance Propagation (LRP) explanations diverge from Leave-One-Out (LOO) attributions in Transformer models. It identifies two main failure modes: bilinear propagation in AttnLRP that violates implementation invariance, and softmax propagation that introduces linearization and structural biases. Analytic counterexamples and MNIST experiments show bilinear rules can yield different attributions for functionally identical factorisations, while CP-LRP often aligns better with LOO, particularly in middle-to-late layers. Layer-wise ablations reveal that bypassing softmax propagation improves LOO alignment in those layers, suggesting that block-wise canonization of attention components could yield more faithful and scalable explanations for Transformers with practical impact on interpretability in high-stakes domains.

Abstract

Leave-One-Out (LOO) provides an intuitive measure of feature importance but is computationally prohibitive. While Layer-Wise Relevance Propagation (LRP) offers a potentially efficient alternative, its axiomatic soundness in modern Transformers remains largely under-examined. In this work, we first show that the bilinear propagation rules used in recent advances of AttnLRP violate the implementation invariance axiom. We prove this analytically and confirm it empirically in linear attention layers. Second, we also revisit CP-LRP as a diagnostic baseline and find that bypassing relevance propagation through the softmax layer -- backpropagating relevance only through the value matrices -- significantly improves alignment with LOO, particularly in middle-to-late Transformer layers. Overall, our results suggest that (i) bilinear factorization sensitivity and (ii) softmax propagation error potentially jointly undermine LRP's ability to approximate LOO in Transformers.

When LRP Diverges from Leave-One-Out in Transformers

TL;DR

This paper analyzes why Layer-Wise Relevance Propagation (LRP) explanations diverge from Leave-One-Out (LOO) attributions in Transformer models. It identifies two main failure modes: bilinear propagation in AttnLRP that violates implementation invariance, and softmax propagation that introduces linearization and structural biases. Analytic counterexamples and MNIST experiments show bilinear rules can yield different attributions for functionally identical factorisations, while CP-LRP often aligns better with LOO, particularly in middle-to-late layers. Layer-wise ablations reveal that bypassing softmax propagation improves LOO alignment in those layers, suggesting that block-wise canonization of attention components could yield more faithful and scalable explanations for Transformers with practical impact on interpretability in high-stakes domains.

Abstract

Leave-One-Out (LOO) provides an intuitive measure of feature importance but is computationally prohibitive. While Layer-Wise Relevance Propagation (LRP) offers a potentially efficient alternative, its axiomatic soundness in modern Transformers remains largely under-examined. In this work, we first show that the bilinear propagation rules used in recent advances of AttnLRP violate the implementation invariance axiom. We prove this analytically and confirm it empirically in linear attention layers. Second, we also revisit CP-LRP as a diagnostic baseline and find that bypassing relevance propagation through the softmax layer -- backpropagating relevance only through the value matrices -- significantly improves alignment with LOO, particularly in middle-to-late Transformer layers. Overall, our results suggest that (i) bilinear factorization sensitivity and (ii) softmax propagation error potentially jointly undermine LRP's ability to approximate LOO in Transformers.
Paper Structure (41 sections, 10 equations, 5 figures, 3 tables)

This paper contains 41 sections, 10 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Error in Bilinear Layers.
  3. Error in Softmax Layers.
  4. Background
  5. Leave-One-Out (LOO) Attribution
  6. Heuristic LOO approximations
  7. A Principled LOO Alternative: Layer-Wise Relevance Propagation (LRP)
  8. Epsilon Rule for Linear Layers.
  9. Rules for Bilinear Layers.
  10. Rule for the Softmax Layer.
  11. Why does LRP still approximate LOO poorly?
  12. Part 1: Bilinear Propagation in LRP Violates Implementation Invariance
  13. Model A Derivation
  14. Model B Derivation
  15. Part 2: Softmax Propagation as a Second Source of Error
  16. ...and 26 more sections

Figures (5)

  • Figure 1: Two functionally equivalent network factorizations of the same function $y = x_1 x_2 x_3$. Despite identical outputs, LRP's $\varepsilon$-rule assigns different relevance to inputs depending on factorization.
  • Figure 2: SST LOO $r$
  • Figure 3: IMDB LOO $r$
  • Figure 4: SST $\Delta=\text{LeRF}-\text{MoRF}$
  • Figure 5: IMDB $\Delta=\text{LeRF}-\text{MoRF}$