Revisiting [CLS] and Patch Token Interaction in Vision Transformers

TL;DR

This work identifies and addresses a friction between global and local representations in Vision Transformers by showing that pre-attention LayerNorm and subsequent processing already differentiate CLS and patch tokens. It introduces layer specialization that uses separate parameters for CLS and patch streams, preserving attention interactions while improving dense prediction tasks. The proposed normalization and projection specialization yields segmentation gains up to around 2.2 mIoU points with roughly 8% more parameters and no inference cost, and demonstrates generalization across DINOv2 variants, ViT sizes, and DeiT-III training. The findings highlight the value of targeted architectural adaptations to enhance dense feature learning without sacrificing global performance, offering a practical route to stronger vision models across tasks like segmentation and depth estimation.

Abstract

Vision Transformers have emerged as powerful, scalable and versatile representation learners. To capture both global and local features, a learnable [CLS] class token is typically prepended to the input sequence of patch tokens. Despite their distinct nature, both token types are processed identically throughout the model. In this work, we investigate the friction between global and local feature learning under different pre-training strategies by analyzing the interactions between class and patch tokens. Our analysis reveals that standard normalization layers introduce an implicit differentiation between these token types. Building on this insight, we propose specialized processing paths that selectively disentangle the computational flow of class and patch tokens, particularly within normalization layers and early query-key-value projections. This targeted specialization leads to significantly improved patch representation quality for dense prediction tasks. Our experiments demonstrate segmentation performance gains of over 2 mIoU points on standard benchmarks, while maintaining strong classification accuracy. The proposed modifications introduce only an 8% increase in parameters, with no additional computational overhead. Through comprehensive ablations, we provide insights into which architectural components benefit most from specialization and how our approach generalizes across model scales and learning frameworks.

Figures (16)

• Figure 1: Visualization of the impact of our proposed layer specialization for [CLS] and patch tokens on the patch features obtained with DINOv2 when using two strategies to mitigate artifacts, namely registers ('regs') darcet2023vision and attention bias ('attn. bias') an2025systematic. We display the first PCA components of model outputs in RGB.
• Figure 2: [CLS]-patches separation effect within transformer blocks in vanilla DINOv2 ViT-L model. We show mean and standard deviation of cosine similarity between [CLS] and all patches, and all-to-all patches, before and after each transformer layers. 'attn.' stands for attention.
• Figure 3: Impact of LayerNorm before attention layer for different pre-trained models. We show mean and standard deviation of cosine similarity between [CLS] and all patches, and between all patches. Statistics visualized before and after LayerNorm (LN).
• Figure 4: Dimensions with biggest magnitudes early (a), in the middle (b), at the end (c) of the model for [CLS] and $5$ patches with the highest magnitudes in the selected dimensions. Tokens taken at the output of blocks. The considered model is a DINOv2 ViT-L with attention bias.
• Figure 5: Architecture specialization. We investigate how [CLS] and patch tokens can be processed through specialized layers, while preserving their interactions within the attention mechanism.