ChannelExplorer: Exploring Class Separability Through Activation Channel Visualization

TL;DR

ChannelExplorer tackles the challenge of interpreting how activation channels across layers influence class separability in vision models by presenting an interactive visual analytics toolkit. It combines three coordinated views—Activation Distance Scatterplot, Activation Jaccard Similarity, and Activation Heatmap—with channel summarization to enable abstract-to-grounded exploration across layers and architectures. The framework is demonstrated on scenarios including ImageNet class hierarchy discovery, mislabeled image detection, channel contribution analysis, and latent-space localization in Stable Diffusion, with expert evaluation validating its utility. The open-source implementation emphasizes data-driven debugging and model refinement, while also outlining limitations and avenues for extending analysis to non-image-based layers and attention mechanisms.

Abstract

Deep neural networks (DNNs) achieve state-of-the-art performance in many vision tasks, yet understanding their internal behavior remains challenging, particularly how different layers and activation channels contribute to class separability. We introduce ChannelExplorer, an interactive visual analytics tool for analyzing image-based outputs across model layers, emphasizing data-driven insights over architecture analysis for exploring class separability. ChannelExplorer summarizes activations across layers and visualizes them using three primary coordinated views: a Scatterplot View to reveal inter- and intra-class confusion, a Jaccard Similarity View to quantify activation overlap, and a Heatmap View to inspect activation channel patterns. Our technique supports diverse model architectures, including CNNs, GANs, ResNet and Stable Diffusion models. We demonstrate the capabilities of ChannelExplorer through four use-case scenarios: (1) generating class hierarchy in ImageNet, (2) finding mislabeled images, (3) identifying activation channel contributions, and(4) locating latent states' position in Stable Diffusion model. Finally, we evaluate the tool with expert users.

Figures (12)

• Figure 1: (A) and (B) show two ways for image activation analysis. We use (A) as each channel represents the presence of features in the input. Channel activations of two layers are shown in (C) and (D). Layer $i$ identifies basic features (i.e., grass texture, fur texture, edges, etc.), whereas layer $j$ identifies complex features (i.e., dog's head, body, etc.).
• Figure 2: Overall workflow of ChannelExplorer. Users start the analysis with Dataset View (A) to find classes with high confusion. Then the system shows intra-class and inter-class confusions using the two layer-level views (B), and constructs class confusion hierarchy (B1). The Heatmap View shows the activations of each channel (C). Finally, this approach can be iteratively used to refine both the model and the dataset.
• Figure 3: InceptionV3 model visualization using ChannelExplorer.
• Figure 4: Effect of different summarization functions on a scatterplot, Jaccard similarity view, and activation channels. All values are globally normalized for transforming to color-scale. Geometric Threshold function
• Figure 5: Four DR methods applied to two layers' summarizations of activation channels of InceptionV3 CNN model. The last CNN layer (mixed10) shows that UMAP and t-SNE preserved the global distances between classes, whereas MDS and PCA focused more on the local structure and variance.