Open Vocabulary Compositional Explanations for Neuron Alignment

TL;DR

This work tackles the challenge of explaining neuron alignment without relying on human-annotated concept masks by introducing a training-free, open vocabulary framework for vision. The method identifies user-specified concept subsets, generates segmentation masks with open vocabulary models, and derives compositional explanations via a MMESH-guided beam search, enabling explanations at multiple granularities. Experiments show the approach achieves competitive quality against baselines on ADE20K and CUB, and maintains interpretability when human annotations are unavailable, while analyses based on WordNet reveal and mitigate misalignment due to granularity and labeling differences. The framework enables flexible, open-world explanations and iterative refinements, with practical implications for trust, customization, and broader applicability in model interpretability.

Abstract

Neurons are the fundamental building blocks of deep neural networks, and their interconnections allow AI to achieve unprecedented results. Motivated by the goal of understanding how neurons encode information, compositional explanations leverage logical relationships between concepts to express the spatial alignment between neuron activations and human knowledge. However, these explanations rely on human-annotated datasets, restricting their applicability to specific domains and predefined concepts. This paper addresses this limitation by introducing a framework for the vision domain that allows users to probe neurons for arbitrary concepts and datasets. Specifically, the framework leverages masks generated by open vocabulary semantic segmentation to compute open vocabulary compositional explanations. The proposed framework consists of three steps: specifying arbitrary concepts, generating semantic segmentation masks using open vocabulary models, and deriving compositional explanations from these masks. The paper compares the proposed framework with previous methods for computing compositional explanations both in terms of quantitative metrics and human interpretability, analyzes the differences in explanations when shifting from human-annotated data to model-annotated data, and showcases the additional capabilities provided by the framework in terms of flexibility of the explanations with respect to the tasks and properties of interest.

Figures (11)

• Figure 1: Examples of misalignment between human and model-annotated data due to different granularity in annotations (top) and the lack of concepts capturing patterns (bottom) in the concept set.
• Figure 2: Explanations associated with neuron #19 and cluster 4 by our framework using different levels of granularity. In blue are areas of neuron activation within the considered range.
• Figure 3: An example of how iterative refinements of the concept set can improve open vocabulary explanations.
• Figure 4: Examples of highly scored "bird" explanations for precision despite most of the images refer to bird parts (head (top) and wings (bottom)). Participants were asked to evaluate only the unmasked regions of the images.
• Figure 5: Explanations associated with Cluster 5 of neurons from 0 to 2 by the Closed approach bau2020units and our framework. In blue are areas of neuron activation within the considered range.