Visual Concept Connectome (VCC): Open World Concept Discovery and their Interlayer Connections in Deep Models

TL;DR

This work introduces Visual Concept Connectome (VCC), an open-world, unsupervised framework that discovers human-interpretable concepts and their interlayer connections across deep vision models. By combining top-down feature-space segmentation, layer-wise concept discovery, and ITCAV-based interlayer connectivity, VCC yields a DAG whose nodes are layer-specific concepts and whose edges quantify how concepts propagate through the network. The approach is validated across CNNs and transformers, demonstrating insights into hierarchical concept assemblies, architecture-dependent differences, and failure-mode debugging. The method enables multi-resolution interpretation, supports architecture analysis, and offers practical utility for diagnosing mispredictions and guiding debiasing efforts.

Abstract

Understanding what deep network models capture in their learned representations is a fundamental challenge in computer vision. We present a new methodology to understanding such vision models, the Visual Concept Connectome (VCC), which discovers human interpretable concepts and their interlayer connections in a fully unsupervised manner. Our approach simultaneously reveals fine-grained concepts at a layer, connection weightings across all layers and is amendable to global analysis of network structure (e.g., branching pattern of hierarchical concept assemblies). Previous work yielded ways to extract interpretable concepts from single layers and examine their impact on classification, but did not afford multilayer concept analysis across an entire network architecture. Quantitative and qualitative empirical results show the effectiveness of VCCs in the domain of image classification. Also, we leverage VCCs for the application of failure mode debugging to reveal where mistakes arise in deep networks.

Figures (19)

• Figure 1: A Visual Concept Connectome (VCC). At each layer the visual concepts learned by a deep model for a given class are revealed as are the learned interlayer concept connections. For each concept, up to four exemplars are shown as unmasked regions in a $2\times 2$ image. Interlayer concept connections are shown as lines with darker lines indicating larger contributions. Shown is a VCC for every convolutional layer of a VGG16 model szegedy2015going trained on ImageNet deng2009imagenet targeting recognition of class "Tow Truck". A closer visualization of VCC subgraphs reveals interesting compositions occurring at different levels of abstraction corresponding at different depths of the model. At early layers (bottom left), we observe oriented patterns ($C_{1,1}$) and brown color ($C_{1,2}$) composing the concept of green and brown orientation ($C_{2,1}$). Middle layers (right) show the concept of 'wheel on the road' ($C_{9,1}$) being composed of wheels ($C_{8,1}$) and regions of asphalt ($C_{8,2}$). The final layer concepts (top left) show that both foreground objects, e.g. tow trucks ($C_{13,1}$), and background regions, e.g. road, trees, humans, or car being towed ($C_{13,2}$), concepts highly influence the final category ($C_{TowTruck}$).
• Figure 2: The three steps in building a Visual Concept Connectome (VCC). (A) For a given image, $\mathcal{I}^i \in \mathcal{I}$, model, $F$, and layer, $n$, we produce a set of image segments, $\mathbf{M}^i_n \in \mathbf{M}_n$, based on a recursive spatial clustering, $\mathsf{C}^{seg}$ (\ref{['eq:cluster']}), of the features $\textbf{z}_n^i$ conditioned on the clusters from the layer above, $n+1$. We then use (\ref{['eq:rgb_mask']}) to generate a set of masked RGB image segments for each layer, $\mathbf{M}_j$. (B) For a given layer, $j$, we pass the image segments from all images, $\mathbf{M}_j$, through $f_j$ and cluster, $\mathsf{C}^{con}$ (\ref{['eq:concept_cluster']}), these features across the dataset to produce $m_j$ concept centroids, $\{\textbf{q}^1_j,\dotsc,\textbf{q}^{m_j}_j\}$. (C) To measure the contribution of an earlier layer concept, $\textbf{q}^{m_j}_j$, to a later layer concept, $\textbf{q}^{m_l}_l$, we employ our Interlayer Testing with CAV (ITCAV) approach (Sec. \ref{['sec:itcav']}), which uses the Concept Activation Vector (CAV) kim2018interpretability of the earlier concept, $\textbf{V}_{\textbf{q}^{m_j}_j}$ (that points away from random examples, $\mathcal{I}_{rnd}$, but toward concept exemplars, $\mathbf{M}_{\textbf{q}^{m_j}_j}$), and the deeper layer concept, $\textbf{q}^{m_l}_l$.
• Figure 3: Validation of the three VCC method components. (a) Validation of segment proposals. The relative (Rel.) concept segment size compared to entire image for a given layer is plotted against the receptive field (RF) width/height of the same layer. (b) Validation of discovered concepts. For 50 randomly selected ImageNet classes, we discover concepts in four layers of the model. During inference, one randomly selected concept at each layer is suppressed by a factor of $\epsilon$. (c, d) Validation of interlayer concept weights. The unnormalized logit sum (LS) scores, (\ref{['eq:LS']}), for the target class are plotted against the average path strength (APS) scores, (\ref{['eq:APS']}). A positive correlation implies that the ITCAV edge weights connecting a concept to the class are predictive of the model output having a higher probability for that class.
• Figure 4: A VCC for three selected layers of a GoogLeNet model szegedy2015going targeting recognition of class 'Jay'. Darker lines denote stronger connection weights.
• Figure 5: Graph metrics on four layer VCCs comparing CNN vs. transformer architectures and training objectives.