Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection

TL;DR

This work addresses object detection with masked-image-modeling (MIM) pre-trained vanilla ViTs, identifying two key gaps: lack of multi-scale pyramid features and high cost of processing high-resolution inputs. It proposes MimDet, a ConvStem–ViT hybrid backbone that processes only partially observed input sequences during fine-tuning and uses a lightweight MAE-like decoder to reconstruct full features, enabling a ConvNet-ViT pipeline with a pyramidal feature hierarchy. On COCO, MimDet with ViT-Base achieves 51.7 AP^box and 46.1 AP^mask, surpassing Swin Transformer by 2.5 AP and 2.6 AP, and converges 2.8× faster; larger variants reach even higher AP, signaling strong scalability. These results suggest that leveraging robust vanilla ViT representations with minimal, targeted architectural adaptations can outperform conventional hierarchical backbones in object-level tasks, hinting at broader applicability of MAE-pre-trained ViTs for vision beyond image-level understanding.

Abstract

We present an approach to efficiently and effectively adapt a masked image modeling (MIM) pre-trained vanilla Vision Transformer (ViT) for object detection, which is based on our two novel observations: (i) A MIM pre-trained vanilla ViT encoder can work surprisingly well in the challenging object-level recognition scenario even with randomly sampled partial observations, e.g., only 25% $\sim$ 50% of the input embeddings. (ii) In order to construct multi-scale representations for object detection from single-scale ViT, a randomly initialized compact convolutional stem supplants the pre-trained large kernel patchify stem, and its intermediate features can naturally serve as the higher resolution inputs of a feature pyramid network without further upsampling or other manipulations. While the pre-trained ViT is only regarded as the 3$^{rd}$-stage of our detector's backbone instead of the whole feature extractor. This results in a ConvNet-ViT hybrid feature extractor. The proposed detector, named MIMDet, enables a MIM pre-trained vanilla ViT to outperform hierarchical Swin Transformer by 2.5 box AP and 2.6 mask AP on COCO, and achieves better results compared with the previous best adapted vanilla ViT detector using a more modest fine-tuning recipe while converging 2.8$\times$ faster. Code and pre-trained models are available at https://github.com/hustvl/MIMDet.

Figures (3)

• Figure 1: Overview of our MimDet and comparisons with previous representative approaches adapting vanilla ViT for object detection (e.g., benchmarking). In MimDet, a randomly initialized compact convolutional stem (ConvStem) replaces the pre-trained large kernel patchify stem (PatchStem), and the ViT encoder only receives and processes sampled partial input embeddings. The intermediate features of ConvStem can directly serve as the higher resolution inputs for a standard FPN FPN.
• Figure 2: Visualizations and comparisons of some stride-4 backbone and FPN feature maps. The feature maps of benchmarking is obtained from our re-implementation which successfully reproduces its reported results.
• Figure 3: More visualizations and comparisons of some stride-4 backbone and FPN feature maps. The feature maps of benchmarking is obtained from our re-implementation which successfully reproduces its reported results.