From Colors to Classes: Emergence of Concepts in Vision Transformers

TL;DR

The paper investigates how Vision Transformers develop semantic concepts across layers using neuron labeling (via CLIP-Dissect). By analyzing four pretrained ViTs and a CNN, it shows a consistent progression from simple concepts (colors, textures) in early layers to complex concepts (objects, natural elements) in later layers, with increasing concept diversity per layer. Finetuning to downstream tasks reduces the number of concepts and shifts them toward task-relevant categories, sometimes at the expense of previously learned information. The work deepens understanding of ViT representations, highlights the potential for concept-based explainability, and underscores model- and task-dependent shifts that inform transfer learning and robustness considerations.

Abstract

Vision Transformers (ViTs) are increasingly utilized in various computer vision tasks due to their powerful representation capabilities. However, it remains understudied how ViTs process information layer by layer. Numerous studies have shown that convolutional neural networks (CNNs) extract features of increasing complexity throughout their layers, which is crucial for tasks like domain adaptation and transfer learning. ViTs, lacking the same inductive biases as CNNs, can potentially learn global dependencies from the first layers due to their attention mechanisms. Given the increasing importance of ViTs in computer vision, there is a need to improve the layer-wise understanding of ViTs. In this work, we present a novel, layer-wise analysis of concepts encoded in state-of-the-art ViTs using neuron labeling. Our findings reveal that ViTs encode concepts with increasing complexity throughout the network. Early layers primarily encode basic features such as colors and textures, while later layers represent more specific classes, including objects and animals. As the complexity of encoded concepts increases, the number of concepts represented in each layer also rises, reflecting a more diverse and specific set of features. Additionally, different pretraining strategies influence the quantity and category of encoded concepts, with finetuning to specific downstream tasks generally reducing the number of encoded concepts and shifting the concepts to more relevant categories.

Figures (10)

• Figure 1: We analyze how different concepts develop across the layers of vision transformers. Early layers tend to process simpler concepts and images while late layers focus on more complex and diverse concepts.
• Figure 2: Similarity scores for concepts and number of different concepts encoded in each layer.
• Figure 3: Categories of concepts compared across models. Early layers focus mostly on colors while middle layers have a large proportion of neurons assigned to textures and materials while objects and natural elements appear in later layers. The overall trend is similar for all models but small differences can be observed, e.g. CLIP has a higher amount of neurons assigned to activities in late layers.
• Figure 4: Examples of the highest activating images for five neurons after the first, the sixth and the last transformer block in the sup-ViT model. While this only shows a small subset, the images are representative of most highest activating images, also across models, and show how the models react mainly to simple images in early layers and more complex images in later layers. Similar plots for other models can be found in the Appendix \ref{['fig:app_active_images']}.
• Figure 5: Complexity of the five highest activating images for each neuron averaged across layers measured by ICNet feng2023ic9600. All models show an increase in image complexity across layers.