You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection

TL;DR

This work investigates whether a vanilla Vision Transformer pre-trained on ImageNet-1k can be directly repurposed for 2D object detection with minimal spatial priors. It introduces YOLOS, which replaces the CLS token with 100 DET tokens and uses a bipartite matching loss to perform set-based predictions, avoiding 2D feature map reinterpretation. The results show competitive COCO performance (e.g., 42.0 box AP for YOLOS-Base) and reveal the transferability of ViT to dense object detection, while analyzing pretraining regimes, scaling effects, and the role of detection tokens. The findings advocate a vision Transformer-centric pretraining paradigm and provide a benchmark for evaluating ViT pretraining strategies on downstream detection tasks.

Abstract

Can Transformer perform 2D object- and region-level recognition from a pure sequence-to-sequence perspective with minimal knowledge about the 2D spatial structure? To answer this question, we present You Only Look at One Sequence (YOLOS), a series of object detection models based on the vanilla Vision Transformer with the fewest possible modifications, region priors, as well as inductive biases of the target task. We find that YOLOS pre-trained on the mid-sized ImageNet-1k dataset only can already achieve quite competitive performance on the challenging COCO object detection benchmark, e.g., YOLOS-Base directly adopted from BERT-Base architecture can obtain 42.0 box AP on COCO val. We also discuss the impacts as well as limitations of current pre-train schemes and model scaling strategies for Transformer in vision through YOLOS. Code and pre-trained models are available at https://github.com/hustvl/YOLOS.

Figures (5)

• Figure 1: YOLOS architecture overview. "Pat-Tok" refers to $[\mathtt{PATCH}]$ token, which is the embedding of a flattened image patch. "Det-Tok" refers to $[\mathtt{DET}]$ token, which is a learnable embedding for object binding. "PE" refers to positional embedding. During training, YOLOS produces an optimal bipartite matching between predictions from one hundred $[\mathtt{DET}]$ tokens and ground truth objects. During inference, YOLOS directly outputs the final set of predictions in parallel. The figure style is inspired by ViT.
• Figure 2: Visualization of all box predictions on all images from COCO $\mathtt{val}$ split for the first ten $[\mathtt{DET}]$ tokens. Each box prediction is represented as a point with the coordinates of its center normalized by each thumbnail image size. The points are color-coded so that blue points corresponds to small objects, green to medium objects and red to large objects. We observe that each $[\mathtt{DET}]$ token learns to specialize on certain regions and sizes. The visualization style is inspired by DETR.
• Figure 3: The statistics of all ground truth object categories (the red curve) and the statistics of all object category predictions from all $[\mathtt{DET}]$ tokens (the blue curve) on all images from COCO $\mathtt{val}$ split. The error bar of the blue curve represents the variability of the preference of different tokens for a given category, which is small. This suggests that different $[\mathtt{DET}]$ tokens are category insensitive.
• Figure 4: The self-attention map visualization of the $[\mathtt{DET}]$ tokens and the corresponding predictions on the heads of the last layer of two different YOLOS-S models.
• Figure 5: The self-attention map visualization of the $[\mathtt{DET}]$ tokens and the corresponding predictions on the heads of the last layer of two different YOLOS-S models.