ViTAR: Vision Transformer with Any Resolution

TL;DR

This work proposes a novel module for dynamic resolution adjustment, designed with a single Transformer block, specifically to achieve highly efficient incremental token integration and introduces fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions, thereby preventing overfitting to any single training resolution.

Abstract

This paper tackles a significant challenge faced by Vision Transformers (ViTs): their constrained scalability across different image resolutions. Typically, ViTs experience a performance decline when processing resolutions different from those seen during training. Our work introduces two key innovations to address this issue. Firstly, we propose a novel module for dynamic resolution adjustment, designed with a single Transformer block, specifically to achieve highly efficient incremental token integration. Secondly, we introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions, thereby preventing overfitting to any single training resolution. Our resulting model, ViTAR (Vision Transformer with Any Resolution), demonstrates impressive adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and 80.4\% accuracy at a 4032x4032 resolution, all while reducing computational costs. ViTAR also shows strong performance in downstream tasks such as instance and semantic segmentation and can easily combined with self-supervised learning techniques like Masked AutoEncoder. Our work provides a cost-effective solution for enhancing the resolution scalability of ViTs, paving the way for more versatile and efficient high-resolution image processing.

Figures (4)

• Figure 1: Comparison with other models: When the input resolution is larger than 1792, both DeiT-B and ResFormer-B encounter Out-of-Memory (OOM) errors. Annotations represent the models' computational load in terms of FLOPS. The results demonstrate that ViTAR possesses low computational overhead and exceptionally strong resolution generalization capability.
• Figure 2: Top: Overall architecture of ViTAR consists of Adaptive Token Merger (ATM), Fuzzy Positional Encoding (FPE), and standard ViT self-attention layers. Bottom: Left: ATM with shared weights is applicable $m$ times, depending on the input resolution. Right: FPE with the dashed red bounding box to denote the regions from which random points are sampled during training for positional embedding computation.
• Figure 3: Illustration of GridAttention.
• Figure 4: Illustration of grid padding.