Interpreting ResNet-based CLIP via Neuron-Attention Decomposition

TL;DR

This work introduces a neuron-attention decomposition to interpret CLIP-ResNet by decomposing outputs into neuron–attention-head computation paths that reside in the joint image-text space. It demonstrates that neuron–head contributions are effectively rank-1 and sparse, enabling automatic labeling with text and revealing sub-concepts within neurons. The authors apply this framework to training-free semantic segmentation, achieving notable mIoU gains, and to distribution-shift monitoring with strong concept-alignment correlations across datasets. Limitations include focusing on the final layer and ongoing polysemanticity, with future work aimed at broader layer coverage and automated labeling of fine-grained components.

Abstract

We present a novel technique for interpreting the neurons in CLIP-ResNet by decomposing their contributions to the output into individual computation paths. More specifically, we analyze all pairwise combinations of neurons and the following attention heads of CLIP's attention-pooling layer. We find that these neuron-head pairs can be approximated by a single direction in CLIP-ResNet's image-text embedding space. Leveraging this insight, we interpret each neuron-head pair by associating it with text. Additionally, we find that only a sparse set of the neuron-head pairs have a significant contribution to the output value, and that some neuron-head pairs, while polysemantic, represent sub-concepts of their corresponding neurons. We use these observations for two applications. First, we employ the pairs for training-free semantic segmentation, outperforming previous methods for CLIP-ResNet. Second, we utilize the contributions of neuron-head pairs to monitor dataset distribution shifts. Our results demonstrate that examining individual computation paths in neural networks uncovers interpretable units, and that such units can be utilized for downstream tasks.

Figures (12)

• Figure 1: Accuracy for reconstruction from principal components. Unlike neuron representations, neuron-head representations match the baseline accuracy using only one direction for reconstruction.
• Figure 2: Reconstruction from sparse text representations. We compare sparse text decompositions for neurons and neuron-head pairs while varying description set sizes.
• Figure 2: Sparse text-based decomposition examples for \ref{['fig-image-retrieval']} components. We select from the descriptions detailed in \ref{['quant-analysis']} with sparsity $m=64$ and use TextSpan (textspan) to obtain descriptions for each attention head.
• Figure 3: Semantic segmentation performance on PASCAL Context. For fair comparison, we implement MaskCLIP (zhou-maskclip) and $QQ^T + KK^T$ attention on the same slide inference setup we use. SC-CLIP, the current state-of-the-art, uses a ViT backbone.
• Figure 4: Images with largest contribution norm for attention heads, neurons, and neuron-head pairs. We present the top images from ImageNet validation set. Neuron-head pairs correspond to specific subcategory concepts of neurons (e.g., 'butterfly clothing' in row 3) and similar concepts to their neurons (e.g., 'router' for neuron $\#2384$ and 'people' for neuron $\#1300$)