Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Better plain ViT baselines for ImageNet-1k

Lucas Beyer, Xiaohua Zhai, Alexander Kolesnikov

TL;DR

The paper challenges the view that Vision Transformers require heavy regularization for ImageNet-1k by showing that a plain ViT setup can achieve state-of-the-art-ish results with minimal, well-chosen tweaks. By using ViT-S/16 with standard augmentations, a 1024-batch regime, global average pooling, fixed sincos2d position embeddings, RandAugment, and Mixup, the authors demonstrate 76.5% top-1 at 90 epochs (~6.5 hours on TPUv3-8) and 80% top-1 at 300 epochs (under a day), approaching ResNet-50 baselines. Extensive ablations indicate that these small changes cumulatively drive large gains, while avoiding more complex tricks like dropout, distillation, or advanced optimizers. The result is a simple, robust baseline for ViT on ImageNet-1k that facilitates fair comparisons and practical deployment.

Abstract

It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.

Better plain ViT baselines for ImageNet-1k

TL;DR

The paper challenges the view that Vision Transformers require heavy regularization for ImageNet-1k by showing that a plain ViT setup can achieve state-of-the-art-ish results with minimal, well-chosen tweaks. By using ViT-S/16 with standard augmentations, a 1024-batch regime, global average pooling, fixed sincos2d position embeddings, RandAugment, and Mixup, the authors demonstrate 76.5% top-1 at 90 epochs (~6.5 hours on TPUv3-8) and 80% top-1 at 300 epochs (under a day), approaching ResNet-50 baselines. Extensive ablations indicate that these small changes cumulatively drive large gains, while avoiding more complex tricks like dropout, distillation, or advanced optimizers. The result is a simple, robust baseline for ViT on ImageNet-1k that facilitates fair comparisons and practical deployment.

Abstract

It is commonly accepted that the Vision Transformer model requires sophisticated regularization techniques to excel at ImageNet-1k scale data. Surprisingly, we find this is not the case and standard data augmentation is sufficient. This note presents a few minor modifications to the original Vision Transformer (ViT) vanilla training setting that dramatically improve the performance of plain ViT models. Notably, 90 epochs of training surpass 76% top-1 accuracy in under seven hours on a TPUv3-8, similar to the classic ResNet50 baseline, and 300 epochs of training reach 80% in less than one day.
Paper Structure (5 sections, 1 figure, 2 tables)

This paper contains 5 sections, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Experimental setup
  3. Results
  4. Conclusion
  5. big_vision experiment configuration

Figures (1)

  • Figure 1: Comparison of ViT model for this note to state-of-the-art ViT and ResNet models. Left plot demonstrates how performance depends on the total number of epochs, while the right plot uses TPUv3-8 wallclock time to measure compute. We observe that our simple setting is highly competitive, even to the canonical ResNet-50 setups.