Compositional Explanations of Neurons

TL;DR

The paper addresses interpreting neurons in deep networks by learning compositional logical concepts that approximate neuron behavior, enabling analysis across vision and NLP. It introduces a compositional search over a base concept space using operators like And, Or, and Not to build logical forms that maximize an IoU based similarity with neuron activations. Empirical results show neurons can encode meaningful abstractions as well as polysemantic or bias-driven cues; interpretability correlates with task performance differently in vision and NLP, and explanations enable copy-paste adversarial interventions that predictably alter model behavior. This approach provides a transparent framework for diagnosing biases, understanding learned representations, and guiding targeted interventions, with reproducible code and clear limitations discussed.

Abstract

We describe a procedure for explaining neurons in deep representations by identifying compositional logical concepts that closely approximate neuron behavior. Compared to prior work that uses atomic labels as explanations, analyzing neurons compositionally allows us to more precisely and expressively characterize their behavior. We use this procedure to answer several questions on interpretability in models for vision and natural language processing. First, we examine the kinds of abstractions learned by neurons. In image classification, we find that many neurons learn highly abstract but semantically coherent visual concepts, while other polysemantic neurons detect multiple unrelated features; in natural language inference (NLI), neurons learn shallow lexical heuristics from dataset biases. Second, we see whether compositional explanations give us insight into model performance: vision neurons that detect human-interpretable concepts are positively correlated with task performance, while NLI neurons that fire for shallow heuristics are negatively correlated with task performance. Finally, we show how compositional explanations provide an accessible way for end users to produce simple "copy-paste" adversarial examples that change model behavior in predictable ways.

Figures (23)

• Figure 1: Given a set of inputs (a) and scalar neuron activations (b) converted into binary masks (c), we generate an explanation via beam search, starting with an inventory of primitive concepts (d), then incrementally building up more complex logical forms (e). We attempt to maximize the IoU score of an explanation (f); depicted is the IoU of $M_{483}(\mathop{\mathrm{\mathbf{x}}}\nolimits)$ and (water OR river) AND NOT blue.
• Figure 2: Example concepts from the Broden dataset bau2017network, reproduced with permission.
• Figure 3: Distribution of IoU versus max formula length. The line indicates mean IoU. $N = 1$ is equivalent to NetDissect bau2017network; IoU scores steadily increase as max formula length increases.
• Figure 4: NetDissect bau2017network assigns unit 106 the label bullring, but in reality it is detects general sports fields, except football fields, as revealed by the length 3 and length 10 explanations.
• Figure 5: Image classification explanations categorized by semantically coherent abstraction (a--b) and specialization (c), and unrelated polysemanticity (d). For clarity, logical forms are length $N = 3$.