Accelerating Vision Transformers with Adaptive Patch Sizes

TL;DR

Vision Transformers suffer from quadratic self-attention cost with long input sequences when processing high-resolution images. The Adaptive Patch Transformer (APT) introduces content-aware patch sizes by partitioning images with a multi-scale, entropy-based criterion and merging patches into a common embedding via a zero-initialized MLP, enabling fast training and inference. Across ImageNet-scale classification and dense vision tasks (VQA, object detection, semantic segmentation), APT delivers substantial wall-clock speedups (up to ~40–50% for large ViTs and higher resolutions) with negligible or no loss in accuracy, and can converge in as little as 1 epoch when fine-tuning from pretrained checkpoints. By supporting sequence packing and compatibility with window attention, APT generalizes to a broad range of vision problems beyond classification, offering a practical path to more efficient ViT deployment.

Abstract

Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and can be applied to a previously fine-tuned ViT, converging in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30\% faster training and inference in visual QA, object detection, and semantic segmentation.

Figures (10)

• Figure 1: Adaptive Patch Sizing. We present APT, Adaptive Patch Transformers, which significantly accelerate vision transformer training and inference by patchifying images based on their content. Complex regions receive more, smaller tokens, while simpler, homogeneous regions receive fewer.
• Figure 2: APT overview. APT works by measuring the entropy at multiple scales and assigning large patch sizes to low entropy patches. All patches are projected to the same size token embedding, and the reduced size input sequence is passed to the transformer.
• Figure 3: Embedding Different Patch Sizes. The smallest size patches are projected with the patch embedding. Larger patches are both split into their sub-patches and resized; the sub-patches are embedded, aggregated with a convolution layer. These are combined with the resized embedding with a zero-initialized MLP zhang2023controlnet.
• Figure 4: Accuracy vs. Throughput under different compute budgets. Comparison between APT and layer-level merging methods on ViT-L and ViT-H. For a fairer evaluation, we also include their re-implemented Advanced (Adv) versions with FlashAttention, shown with a dashed line. APT consistently outperforms the baselines in both throughput and accuracy across all compute budgets.
• Figure 5: Visualized Examples. APT consistently places large patches on more homogenous regions and smaller patches on more complex ones. We use conservative thresholds to limit information loss. Images are best viewed zoomed in. More visualizations are in Appendix.