Fixed Point Explainability

TL;DR

The paper proposes fixed point explanations as a formal, recursive framework to assess the stability, faithfulness, and minimality of explanations by studying the convergence of an explainer when applied repeatedly to a model. It formalizes the notions of fixed points and $P$-fixed points, and derives convergence guarantees for feature-based, prototype-based, and mechanistic (SAE) explainers, including examples with LLMs. Through extensive experiments on MNIST/FashionMNIST/CIFAR10 and large language models, it shows that fixed-point analysis can reveal inconsistencies, self-consistency failures, and scaling effects on convergence, offering a diagnostic tool for robust explainability. The work highlights both the potential of recursion-based evaluation and the limitations when dealing with non-monotonic components (e.g., certain LLMs) and rule-based systems, pointing to future work on broader classes of explainers and explicit regularization for self-consistency.

Abstract

This paper introduces a formal notion of fixed point explanations, inspired by the "why regress" principle, to assess, through recursive applications, the stability of the interplay between a model and its explainer. Fixed point explanations satisfy properties like minimality, stability, and faithfulness, revealing hidden model behaviours and explanatory weaknesses. We define convergence conditions for several classes of explainers, from feature-based to mechanistic tools like Sparse AutoEncoders, and we report quantitative and qualitative results for several datasets and models, including LLMs such as Llama-3.3-70B.

Figures (10)

• Figure 1: Left: standard explainability pipeline for an image classifier. Right: how our framework derives a fixed point explanation for the classifier on the left.
• Figure 2: A $P$-fixed point LIME explanation for a VGG16 network simonyan2015deepconvolutionalnetworkslargescale on CIFAR10 krizhevsky2009learning. At each iteration, part of the input is occluded (the yellow patches) by the explainer but the model's classification is preserved and correct. After iteration $4$ (the fixed point), the correct classification will hold up to infinity.
• Figure 3: An example of an input that, upon convergence, violates the property of inducing the correct label. The model is a VGG16 network on CIFAR10; the explainer is LRP.
• Figure 4: An example of how a fixed point SHAP explanation can enhance a FashionMNIST xiao2017fashionmnistnovelimagedataset explanation by augmenting the explanation length (for a VGG16 net). While the input is correctly classified at the beginning, the successive refinements violate induce the wrong label for the classifier. The process eventually converges to a very succinct explanation that is nonetheless classified correctly.
• Figure 5: Self-consistency patterns for two prototype-based models. Each image correspond to one decoded prototype, and arrows indicate the nearest prototype according to the model’s similarity function.

Theorems & Definitions (15)

• Definition 2.1: Fixed point explanation
• Definition 2.2: Recursive explanation
• Definition 2.3: Fixed point convergence
• Definition 2.4: A certificate for a fixed point explanations
• Theorem 1
• proof
• Theorem 2
• proof
• Definition 3.1: Self-Consistency
• Corollary 2.1