Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guaranteed Optimal Compositional Explanations for Neurons

Biagio La Rosa, Leilani H. Gilpin

TL;DR

This work tackles the problem of explaining individual neuron activations by aligning them with human concepts through compositional logic, formalized as a maximization of IoU between neuron activations and concept labels. It introduces a decomposed IoU, dIoU, and a suite of quantities that capture how unique and common elements contribute to alignment, along with a heuristic and an optimal best-first search algorithm that guarantees optimal explanations within feasible runtimes. The authors show that standard beam search can be suboptimal (10–40% of cases in high-complexity settings) and demonstrate that a beam-search variant guided by their decomposition and estimates can achieve competitive runtimes with greater flexibility. The framework is designed to be domain-agnostic, with demonstrated applicability in computer vision and CNNs, and the authors provide a path toward broader use in semantic segmentation and other spatially grounded tasks. Overall, this work provides a principled route to provably optimal explanations and offers practical guidance for efficient near-optimal explanations in resource-constrained settings.

Abstract

While neurons are the basic units of deep neural networks, it is still unclear what they learn and if their knowledge is aligned with that of humans. Compositional explanations aim to answer this question by describing the spatial alignment between neuron activations and concepts through logical rules. These logical descriptions are typically computed via a search over all possible concept combinations. Since computing the spatial alignment over the entire state space is computationally infeasible, the literature commonly adopts beam search to restrict the space. However, beam search cannot provide any theoretical guarantees of optimality, and it remains unclear how close current explanations are to the true optimum. In this theoretical paper, we address this gap by introducing the first framework for computing guaranteed optimal compositional explanations. Specifically, we propose: (i) a decomposition that identifies the factors influencing the spatial alignment, (ii) a heuristic to estimate the alignment at any stage of the search, and (iii) the first algorithm that can compute optimal compositional explanations within a feasible time. Using this framework, we analyze the differences between optimal and non-optimal explanations in the most popular settings for compositional explanations, the computer vision domain and Convolutional Neural Networks. In these settings, we demonstrate that 10-40 percent of explanations obtained with beam search are suboptimal when overlapping concepts are involved. Finally, we evaluate a beam-search variant guided by our proposed decomposition and heuristic, showing that it matches or improves runtime over prior methods while offering greater flexibility in hyperparameters and computational resources.

Guaranteed Optimal Compositional Explanations for Neurons

TL;DR

This work tackles the problem of explaining individual neuron activations by aligning them with human concepts through compositional logic, formalized as a maximization of IoU between neuron activations and concept labels. It introduces a decomposed IoU, dIoU, and a suite of quantities that capture how unique and common elements contribute to alignment, along with a heuristic and an optimal best-first search algorithm that guarantees optimal explanations within feasible runtimes. The authors show that standard beam search can be suboptimal (10–40% of cases in high-complexity settings) and demonstrate that a beam-search variant guided by their decomposition and estimates can achieve competitive runtimes with greater flexibility. The framework is designed to be domain-agnostic, with demonstrated applicability in computer vision and CNNs, and the authors provide a path toward broader use in semantic segmentation and other spatially grounded tasks. Overall, this work provides a principled route to provably optimal explanations and offers practical guidance for efficient near-optimal explanations in resource-constrained settings.

Abstract

While neurons are the basic units of deep neural networks, it is still unclear what they learn and if their knowledge is aligned with that of humans. Compositional explanations aim to answer this question by describing the spatial alignment between neuron activations and concepts through logical rules. These logical descriptions are typically computed via a search over all possible concept combinations. Since computing the spatial alignment over the entire state space is computationally infeasible, the literature commonly adopts beam search to restrict the space. However, beam search cannot provide any theoretical guarantees of optimality, and it remains unclear how close current explanations are to the true optimum. In this theoretical paper, we address this gap by introducing the first framework for computing guaranteed optimal compositional explanations. Specifically, we propose: (i) a decomposition that identifies the factors influencing the spatial alignment, (ii) a heuristic to estimate the alignment at any stage of the search, and (iii) the first algorithm that can compute optimal compositional explanations within a feasible time. Using this framework, we analyze the differences between optimal and non-optimal explanations in the most popular settings for compositional explanations, the computer vision domain and Convolutional Neural Networks. In these settings, we demonstrate that 10-40 percent of explanations obtained with beam search are suboptimal when overlapping concepts are involved. Finally, we evaluate a beam-search variant guided by our proposed decomposition and heuristic, showing that it matches or improves runtime over prior methods while offering greater flexibility in hyperparameters and computational resources.

Paper Structure

This paper contains 52 sections, 1 theorem, 93 equations, 10 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Optimal Compositional Explanations
  3. Background and Related Work
  4. Preliminaries and Terminology
  5. Fundamental Quantities
  6. Heuristic
  7. Estimate Label Quantities
  8. Aggregated Computation
  9. Estimate Paths
  10. Aggregated Computation
  11. Optimal Algorithm
  12. Analysis
  13. Feasibility of Optimality and Heuristic-Guided Beam Search
  14. Explanation Analysis
  15. Algorithm Analysis: Insights and Limitations
  16. ...and 37 more sections

Key Result

Lemma 1

Given a binary neuron activation matrix ${\bm{N}}$ and a label $L \in \mathfrak{L^n}$, the decomposed alignment between the annotations associated with $L$ and the neuron activations is equivalent to the Intersection Over Union if the logical operators involved in $L$ are 00-preserving:

Figures (10)

  • Figure 1: An example of a case where the optimal algorithm finds a combination of concepts expressing the highest spatial alignment that beam search fails to capture.
  • Figure 2: Alignment detected in units of a ResNet18 model by both the optimal and beam-search methods. The blue regions indicate areas of neuron activation within the considered range.
  • Figure 3: Alignment detected in units of a ResNet18 model by both the optimal and beam-search methods. The blue regions indicate areas of neuron activation within the considered range.
  • Figure 4: Alignment detected in units of a ResNet18 model by both the optimal and beam-search methods. The blue regions indicate areas of neuron activation within the considered range.
  • Figure 5: Alignment detected in units of an AlexNet model by both the optimal and beam-search methods. The blue regions indicate areas of neuron activation within the considered range.
  • ...and 5 more figures

Theorems & Definitions (8)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Definition 2.4
  • Definition 2.5
  • Lemma 1
  • proof
  • proof