NeurFlow: Interpreting Neural Networks through Neuron Groups and Functional Interactions

TL;DR

NeurFlow addresses the interpretability challenge of deep neural networks by shifting focus from individual neurons to groups of neurons and their functional interactions. It defines core concept neurons, clusters them into semantic groups, and constructs a hierarchical neuron circuit with edge weights computed via Integrated Gradients, enabling cross-layer explanations without predefined concepts. The framework is validated on ImageNet-classic CNNs, demonstrating that core concept neurons are impactful and that group-level interactions faithfully reflect the network's reasoning. It also enables automated image debugging and layer-by-layer relation labeling via multimodal LLMs, offering scalable, automated explanations with practical applicability. Overall, NeurFlow reduces interpretability complexity while delivering faithful, actionable insights into neural computation.

Abstract

Understanding the inner workings of neural networks is essential for enhancing model performance and interpretability. Current research predominantly focuses on examining the connection between individual neurons and the model's final predictions. Which suffers from challenges in interpreting the internal workings of the model, particularly when neurons encode multiple unrelated features. In this paper, we propose a novel framework that transitions the focus from analyzing individual neurons to investigating groups of neurons, shifting the emphasis from neuron-output relationships to functional interaction between neurons. Our automated framework, NeurFlow, first identifies core neurons and clusters them into groups based on shared functional relationships, enabling a more coherent and interpretable view of the network's internal processes. This approach facilitates the construction of a hierarchical circuit representing neuron interactions across layers, thus improving interpretability while reducing computational costs. Our extensive empirical studies validate the fidelity of our proposed NeurFlow. Additionally, we showcase its utility in practical applications such as image debugging and automatic concept labeling, thereby highlighting its potential to advance the field of neural network explainability.

Figures (18)

• Figure 1: Workflow of NeurFlow, consisting of three main components: identifying core concept neurons in each layer, building the neuron circuit, and constructing the circuit of neuron groups.
• Figure 2: The interaction between a neuron $s_i$ and its parent $a$.
• Figure 3: Illustration of our algorithm to determine groups of neurons.
• Figure 4: The difference in losses between core concept neurons and random neuron combinations. The blue-toned bars represent the average losses, while the pink-toned bars indicate instances where random neuron combinations result in smaller losses compared to core concept neurons.
• Figure 5: Effects of neuron groups on model's performance. Retaining only the core concept (denoted as CC) neurons preserves high accuracy, whereas masking them leads to a significant drop in performance. In contrast, random neuron combinations show the opposite trend.

Theorems & Definitions (3)

• Definition 1: Neuron Concept
• Definition 2: Neuron Concept with Knockout
• Definition 3: Core Concept Neurons