Understanding Distributed Representations of Concepts in Deep Neural Networks without Supervision

TL;DR

The paper tackles interpreting deep neural networks without supervision by introducing an unsupervised framework that identifies distributed concept representations via activation configurations. It defines a Configuration distance $d_C$ as the Hamming distance over neuron activations on a chosen subset $N$, and constructs a Relaxed Decision Region (RDR) by selecting a principal subset $N^*$ of size $t$ and a principal configuration $c_p$, maximizing coherence with a positive set $S$ while discriminating from a negative set $S_{neg}$ using a greedy optimization. The approach enables automatic discovery of learned concepts, including unlabeled subclasses and misclassification causes, and reveals layer-wise concept representations across different depths. Empirical results across six datasets and multiple architectures show that RDR yields coherent, interpretable concept groups, aligns with human-labeled Broden concepts, supports misclassification reasoning and debugging, and generalizes across layers and models for practical interpretability.

Abstract

Understanding intermediate representations of the concepts learned by deep learning classifiers is indispensable for interpreting general model behaviors. Existing approaches to reveal learned concepts often rely on human supervision, such as pre-defined concept sets or segmentation processes. In this paper, we propose a novel unsupervised method for discovering distributed representations of concepts by selecting a principal subset of neurons. Our empirical findings demonstrate that instances with similar neuron activation states tend to share coherent concepts. Based on the observations, the proposed method selects principal neurons that construct an interpretable region, namely a Relaxed Decision Region (RDR), encompassing instances with coherent concepts in the feature space. It can be utilized to identify unlabeled subclasses within data and to detect the causes of misclassifications. Furthermore, the applicability of our method across various layers discloses distinct distributed representations over the layers, which provides deeper insights into the internal mechanisms of the deep learning model.

Figures (25)

• Figure 1: Relaxed Decision Region (RDR). Top: Description of the target sample and its neighbors. Middle: Visualization of the feature space and the RDR. Our RDR framework groups instances that have similar neuron activation states in the feature space. Bottom: Instances in the RDR share the coherent concept of 'a person with a stick'.
• Figure 2: Without prior knowledge of label information, our RDR framework successfully captures learned concepts such as subclasses, shapes, crowds, composition, and the degree of flowering, as well as simple color schemes.
• Figure 3: Top 4 nearest instances with different distance metrics. In the case of the Euclidean distance and the Cosine distance, the irrelevant instances are detected. These instances have large Configuration distances from the target.
• Figure 4: Mapping differences with 30 nearest neighbors. The Configuration distance indeed captures instances whose mappings are close to the target's one. With a smaller mapping difference, the image is more similar to the target.
• Figure 5: Retrieval results using RDR and masking for Concept Location. Without supervision, RDR successfully groups similar instances and related parts. The concept of 'legs' is learned for the images in the 12th layer.

Theorems & Definitions (8)

• Definition 1: Activation State
• Definition 2: Configuration
• Definition 3: Configuration Distance
• Theorem 1
• proof
• Lemma 2
• proof
• proof : proof of Theorem 1