Aligned explanations in neural networks

TL;DR

The paper addresses the misalignment between explanations and neural-network decisions by proposing explanatory alignment as a design principle and introducing model readability. It formalizes this via a pseudo-linear framework (PiNets) in which predictions are computed as instance-wise linear functions on a user-defined feature space $\mathcal{Z}$, with explanations given by coefficients $\boldsymbol{\pi}(\mathbf{x})$ produced by an encoder-decoder pair and a second look $\boldsymbol{\pi}(\mathbf{x}) \circ \mathbf{z}$. Key contributions include defining explanatory alignment, establishing MARS faithfulness criteria (Meaningful, Aligned, Robust, Sufficient), and demonstrating PiNets across image classification and segmentation tasks, with techniques like recursive feedback, ensembling, and strong supervision enhancing explanation fidelity. The results show PiNets can match or surpass Grad-CAMs under optimal detection while maintaining accuracy, and can leverage weak labels (e.g., image-level targets) to improve explanations, offering practical benefits for real-world applications and broader data modalities.

Abstract

Feature attribution is the dominant paradigm for explaining deep neural networks. However, most existing methods only loosely reflect the model's prediction-making process, thereby merely white-painting the black box. We argue that explanatory alignment is a key aspect of trustworthiness in prediction tasks: explanations must be directly linked to predictions, rather than serving as post-hoc rationalizations. We present model readability as a design principle enabling alignment, and PiNets as a modeling framework to pursue it in a deep learning context. PiNets are pseudo-linear networks that produce instance-wise linear predictions in an arbitrary feature space, making them linearly readable. We illustrate their use on image classification and segmentation tasks, demonstrating how PiNets produce explanations that are faithful across multiple criteria in addition to alignment.

Figures (9)

• Figure 1: Generic architecture of a PiNet (left) and examples of explanation in the ToyShapes task (right). Column headings indicate the detection strategy (naive or optimal) and the test detection score (defined in eq. \ref{['eq:score']}).
• Figure 2: MARS criteria. An explanation is meaningful if it explains the prediction with relevant signal, aligned if it directly underlies the prediction, robust if it does not heavily rely on context, and sufficient if the prediction can be recovered from it.
• Figure 3: PiNet with recursive feedback. The explanation is used to construct the recursive input $\boldsymbol{\pi}(\mathbf{x})\circ\mathbf{z}$. The discrepancy between the initial explanation $\boldsymbol{\pi}(\mathbf{x})$ and the recursive explanation $\boldsymbol{\pi}'(\mathbf{x})$ is penalized.
• Figure 4: Distribution of meaningfulness in ToyShapes depicted by violin plots. Marks inside each violin represent medians. Blue vertical lines extend the medians of the baseline (Grad-CAMs). "Naive" and "Optimal" refer to the detection strategy.
• Figure 5: Meaningfulness under the optimal detection strategy (thresholding) in ToyShapes. Red contours show iso-levels of the composite score (eq. \ref{['eq:score']}). Points represent fits color-coded by models, and are shown together with 95% Gaussian-confidence ellipses. Ellipses inside the gray insets (left and middle panels) are shown without points for visual clarity; middle and right panels magnify such insets showing the points.