Register and [CLS] tokens yield a decoupling of local and global features in large ViTs

TL;DR

This work challenges the patch integration assumption for Vision Transformers by revealing that, in large models, global information is stored in register tokens and within the CLS pathway via residual connections, producing cleaner attention maps but a decoupled, often misleading relationship between local patches and global representations. The authors demonstrate that as model size grows, the global output is increasingly dominated by information from register tokens or skip connections, undermining the notion that the CLS-attention is a faithful convex combination of patch features. Through analyses like centered kernel alignment and attention ablations, they show that patch-based representations fail to faithfully capture the true global embedding in giant models, and that interpretable attention maps do not reliably ground global behavior in local patches. They propose design guidance—avoiding register tokens and CLS skip mechanisms—to preserve alignment between local and global features and improve interpretability. The findings have implications for dense prediction tasks and the broader interpretability of foundation vision models, motivating new regularization and architectural choices that balance performance with faithful patch-global grounding.

Abstract

Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the [CLS] token itself leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behavior to register and [CLS] tokens, we show a path towards more interpretable vision models.

Figures (9)

• Figure 1: The global image representation computed by a Vision Transformer is usually understood as a weighted average of the local patch features, where weights are given by the [CLS] token attention scores. This notion, which we refer to as the patch integration assumption, underlies attribution methods like plotting the attention maps to identify patches that strongly contribute to the global output. In this work, we show that both register tokens and the [CLS] token lead to violations of this assumption in large ViT models.
• Figure 2: a) The amount of attention placed by the [CLS] token of the last layer onto the patch and register tokens, respectively. Smaller models attend primarily to the patch tokens, whereas bigger models attend more strongly to the register tokens. b) Mean activations of the highest-norm register token in the last layer of the 'giant' model. The 100 dimensions with highest activations before the layer norm are shown. Register tokens show large activations in a small subspace, making them seemingly image-independent as measured by pairwise cosine similarity (top panel). However, the layer norm downscales these dimensions before the self-attention mechanism (bottom panel).
• Figure 3: a) Centered kernel alignment between the global [CLS] token output, and [CLS] token output computed while only attending to either register tokens or patch tokens. Patch tokens yield a faithful representation of the global output for smaller models, but the connection between local and global features breaks down with increasing model size. b) One-shot classification accuracy on the 1000 Imagenet classes. The classifier is trained on the global [CLS] output and then tested on output based on patch and register tokens, respectively. Attending only to the patch tokens yields poor performance in the larger models, corroborating the finding that global representations are not formed by attending to the patch features.
• Figure 4: Attention maps of the final [CLS] token. A convex combination of the corresponding patch features yields the patch-based contribution to the global image representation. As noted by darcetVisionTransformersNeed2024, the attention map of the DINOv2 model exhibits large artifacts. These are removed by including register tokens in the model, seemingly leading to a more interpretable attention map. However, when computing the [CLS] output based on the convex combination of patch features in the model with registers, its cosine similarity to the total output of the final layer is -0.0092. In other words, attending to the patch tokens yields a representation completely orthogonal to the one including the register tokens, showing that the attention map fails to attribute global information to image patches.
• Figure 5: a) After the self-attention mechanism, a skip connection sums the attention output and the hidden states from the previous layer, providing an alternative way for the [CLS] token to attend to itself. Since the attention weights are given only implicitly, we plot the $L_2$-norm of the contributions of the skip connections and the patch features to the [CLS] token as a proxy. We observe that in the 'giant' model, the global output is primarily determined by previously computed features, rather than the patch features. b) We show the cosine similarity of the [CLS] token at all model layers to the [CLS] token at the last layer. The three smaller models exhibit a large jump at the very last layer, indicating that the [CLS] token at the last layer is strongly influenced by the patch tokens. Conversely, the [CLS] token of the 'giant' model converges to the final output more smoothly, explaining its low reliance on the patch features at the last layer.