Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

TL;DR

The paper establishes a standardized, large-scale benchmark to compare efficiency across 45+ Vision Transformer variants, evaluating accuracy, speed, and memory under a unified training pipeline and analyzing results via Pareto fronts. It demonstrates that ViT remains Pareto-optimal across multiple metrics and that scaling model size often yields greater efficiency than increasing image resolution, while hybrid CNN–attention and token-sequence strategies offer niche gains in memory or speed. The work provides a comprehensive taxonomy of efficiency-improving changes, empirical guidance for practitioners on selecting transformers under different constraints, and a centralized resource to accelerate progress in efficient vision transformers. The empirical, hardware-consistent measurements emphasize that theoretical FLOPs or parameter counts alone are insufficient to predict real-world performance.

Abstract

Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity, but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However, diverse experimental conditions, spanning multiple input domains, prevent a fair comparison based solely on reported results, posing challenges for model selection. To address this gap in comparability, we perform a large-scale benchmark of more than 45 models for image classification, evaluating key efficiency aspects, including accuracy, speed, and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front -- the boundary of optimal models. Surprisingly, despite claims of other models being more efficient, ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover, our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation, we provide a centralized resource for practitioners and researchers, facilitating informed decisions when selecting or developing efficient transformers.

Figures (21)

• Figure 1: Pareto front (dotted line) showing the throughput-accuracy trade-off for efficient Vision Transformers. Markers (varying in shape and hue) represent different efficiency strategies, with black dots highlighting Pareto optimal models. Marker size indicates fine-tuning resolution. Baselines ViT-Ti@224 (A), ViT-Ti@384 (B), and ViT-S@224 (C) are shown, along with the Pareto front for high-resolution images (dashed line).
• Figure 2: List of efficient Transformers (citation key in brackets) categorized at two levels: 1. Where does the approach change ViT? 2. How does the approach change ViT?
• Figure 3: Accuracy (red line, right y-axis) and accuracy per parameter (bars, left y-axis) of models ordered by accuracy at a resolution of $224$px. 14 of 47 models of intermediate size are grouped into others. See \ref{['apdx:acc-per-param']} for the full plot.
• Figure 4: Pareto front of finetuning time and accuracy for models which need less than 50 hours for finetuning. We include the full plot in \ref{['apdx:finetuning-time']}.
• Figure 5: Pareto front (dotted line) of training memory at our default batch size of $2048$ (left) and inference memory at the minimum batch size of $1$ (right) and accuracy for models with less than 225GB of VRAM for training and 1.25GB for inference.