Interpreting the Second-Order Effects of Neurons in CLIP

TL;DR

This work introduces a second-order lens to interpret CLIP neurons by tracing their contributions through downstream attention heads to the output, revealing that neuron effects are highly selective and concentrated in late layers. By decomposing these second-order directions into sparse text representations, the authors uncover polysemantic neuron behavior and leverage it for automated semantic adversarial image generation and improved zero-shot segmentation. The approach demonstrates both the risks of automated neuron interpretation (deception) and its potential to create new capabilities in model understanding and manipulation. Overall, the paper provides a scalable framework for mechanistic interpretability with practical implications for robustness and safety in multimodal models.

Abstract

We interpret the function of individual neurons in CLIP by automatically describing them using text. Analyzing the direct effects (i.e. the flow from a neuron through the residual stream to the output) or the indirect effects (overall contribution) fails to capture the neurons' function in CLIP. Therefore, we present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output. We find that these effects are highly selective: for each neuron, the effect is significant for <2% of the images. Moreover, each effect can be approximated by a single direction in the text-image space of CLIP. We describe neurons by decomposing these directions into sparse sets of text representations. The sets reveal polysemantic behavior - each neuron corresponds to multiple, often unrelated, concepts (e.g. ships and cars). Exploiting this neuron polysemy, we mass-produce "semantic" adversarial examples by generating images with concepts spuriously correlated to the incorrect class. Additionally, we use the second-order effects for zero-shot segmentation, outperforming previous methods. Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.

Figures (16)

• Figure 1: Second order effects of CLIP's neurons. Top: We analyze the second-order effects of neurons in CLIP-ViT (flow in pink). Bottom-left: Each second-order effect of a neuron can be decomposed to a sparse set of word directions in the joint text-image space. Bottom-right: co-appearing words in these sets can be used for mass-generation of semantic adversarial images.
• Figure 2: First/Second-order effects. The first order is the flow coming from a neuron to the projection layer and the output (blue). The second order goes from a single neuron through all the consecutive attention heads, to the projection layer, and to the output (pink).
• Figure 3: Mean-ablation of second order effects (ViT-B-32). We evaluate the performance on ImageNet validation set. Second-order effects concentrate in late layers, significant for only a part of the images, and can be approximated by one direction in the output space.
• Figure 4: Comparison to indirect effect. We compare the second-order effects and the indirect effects by mean-ablating layer 9 in ViT-B-32 on ImageNet validation set.
• Figure 5: Accuracy for neuron reconstructed from sparse text representations (ViT-B-32, layer 9). We evaluate the sparse text decompositions for different initial description pools and description set sizes.