Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ViT Registers and Fractal ViT

Jason Chuan-Chih Chou, Abhinav Kumar, Shivank Garg

TL;DR

The paper investigates whether mask-based positional cues can enhance Vision Transformers by introducing input-independent tokens (registers) and fractal summary tokens with a hierarchical attention mask. It formalizes fractal ViT by adding $k\times k$ summary tokens and a global token, then compares various positional encodings (e.g., sincos2d, learned, 2D-ALiBi) across ablations. Across ImageNet-1k with ViT-S/16, sincos2d consistently performs best, while fractal masking provides no improvement over the register baseline and extra-token encodings are largely inconsequential. The results challenge the generality of NoPE-like masking in ViT and motivate reexamining such masking in LM contexts and exploring symmetry-aware encodings in domain-specific settings.

Abstract

Drawing inspiration from recent findings including surprisingly decent performance of transformers without positional encoding (NoPE) in the domain of language models and how registers (additional throwaway tokens not tied to input) may improve the performance of large vision transformers (ViTs), we invent and test a variant of ViT called fractal ViT that breaks permutation invariance among the tokens by applying an attention mask between the regular tokens and ``summary tokens'' similar to registers, in isolation or in combination with various positional encodings. These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.

ViT Registers and Fractal ViT

TL;DR

The paper investigates whether mask-based positional cues can enhance Vision Transformers by introducing input-independent tokens (registers) and fractal summary tokens with a hierarchical attention mask. It formalizes fractal ViT by adding summary tokens and a global token, then compares various positional encodings (e.g., sincos2d, learned, 2D-ALiBi) across ablations. Across ImageNet-1k with ViT-S/16, sincos2d consistently performs best, while fractal masking provides no improvement over the register baseline and extra-token encodings are largely inconsequential. The results challenge the generality of NoPE-like masking in ViT and motivate reexamining such masking in LM contexts and exploring symmetry-aware encodings in domain-specific settings.

Abstract

Drawing inspiration from recent findings including surprisingly decent performance of transformers without positional encoding (NoPE) in the domain of language models and how registers (additional throwaway tokens not tied to input) may improve the performance of large vision transformers (ViTs), we invent and test a variant of ViT called fractal ViT that breaks permutation invariance among the tokens by applying an attention mask between the regular tokens and ``summary tokens'' similar to registers, in isolation or in combination with various positional encodings. These models do not improve upon ViT with registers, highlighting the fact that these findings may be scale, domain, or application-specific.
Paper Structure (12 sections, 6 equations, 1 figure, 2 tables)

This paper contains 12 sections, 6 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Positional Encoding
  4. Attention Patterns
  5. Methodology
  6. Results
  7. Conclusion
  8. 4-summary mask snippet
  9. Positional Encoding
  10. ALiBi: Position-Aware Attention Mechanism
  11. 2D ALiBi Extension
  12. Sinusoidal 2D Positional Embeddings

Figures (1)

  • Figure 1: Fractal ViT architecture. Left: Attention diagram. For clarity, only the attention among different types of tokens at the lower-right corner is drawn as arrows. The toy example shown here uses 2-summary that assigns a summary token for every $2\times2$ regular tokens. Right: Regular tokens created from linear projection of RGB values of patches are fed to the transformer encoder along with zero-init summary tokens and global token, optionally after adding the positional encoding.