Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ConvMAE: Masked Convolution Meets Masked Autoencoders

Peng Gao, Teli Ma, Hongsheng Li, Ziyi Lin, Jifeng Dai, Yu Qiao

TL;DR

ConvMAE introduces a hybrid convolution–transformer encoder with block-wise masking and masked convolutions to enable efficient masked autoencoding with multi-scale representations. The framework yields discriminative, hierarchical features that improve ImageNet finetuning, COCO object detection, ADE20K segmentation, and video understanding (VideoConvMAE), often with shorter pretraining or fewer visible tokens than MAE. Key contributions include the block-wise masking strategy, masked convolutions to prevent information leakage, and a multi-scale decoder that supervises multi-resolution features. The approach demonstrates strong cross-domain gains and faster convergence, highlighting the value of integrating local and global inductive biases in self-supervised vision models.

Abstract

Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained ConvMAE models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at https://github.com/Alpha-VL/ConvMAE.

ConvMAE: Masked Convolution Meets Masked Autoencoders

TL;DR

ConvMAE introduces a hybrid convolution–transformer encoder with block-wise masking and masked convolutions to enable efficient masked autoencoding with multi-scale representations. The framework yields discriminative, hierarchical features that improve ImageNet finetuning, COCO object detection, ADE20K segmentation, and video understanding (VideoConvMAE), often with shorter pretraining or fewer visible tokens than MAE. Key contributions include the block-wise masking strategy, masked convolutions to prevent information leakage, and a multi-scale decoder that supervises multi-resolution features. The approach demonstrates strong cross-domain gains and faster convergence, highlighting the value of integrating local and global inductive biases in self-supervised vision models.

Abstract

Vision Transformers (ViT) become widely-adopted architectures for various vision tasks. Masked auto-encoding for feature pretraining and multi-scale hybrid convolution-transformer architectures can further unleash the potentials of ViT, leading to state-of-the-art performances on image classification, detection and semantic segmentation. In this paper, our ConvMAE framework demonstrates that multi-scale hybrid convolution-transformer can learn more discriminative representations via the mask auto-encoding scheme. However, directly using the original masking strategy leads to the heavy computational cost and pretraining-finetuning discrepancy. To tackle the issue, we adopt the masked convolution to prevent information leakage in the convolution blocks. A simple block-wise masking strategy is proposed to ensure computational efficiency. We also propose to more directly supervise the multi-scale features of the encoder to boost multi-scale features. Based on our pretrained ConvMAE models, ConvMAE-Base improves ImageNet-1K finetuning accuracy by 1.4% compared with MAE-Base. On object detection, ConvMAE-Base finetuned for only 25 epochs surpasses MAE-Base fined-tuned for 100 epochs by 2.9% box AP and 2.2% mask AP respectively. Code and pretrained models are available at https://github.com/Alpha-VL/ConvMAE.
Paper Structure (15 sections, 1 equation, 4 figures, 8 tables)

This paper contains 15 sections, 1 equation, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. A Brief Revisit of MAE
  4. ConvMAE
  5. ConvMAE for Object Detection and Semantic Segmentation
  6. ConvMAE for Video Understanding
  7. Experiments
  8. ImageNet-1K Pretraining and Finetuning
  9. Object Detection
  10. Semantic Segmentation
  11. Video Understanding
  12. Ablation Study of ConvMAE
  13. Related Work
  14. Conclusion
  15. Appendix

Figures (4)

  • Figure 1: The pipeline of our proposed ConvMAE which consists of a hybrid convolution-transformer encoder, block-wise masking strategy with masked convolution and multi-scale decoder.
  • Figure 2: Overview of finetuning ConvMAE for object detection and semantic segmentation. The intermediate features of different stages serve as multi-scale inputs for an FPN lin2017feature module.
  • Figure 3: Finetuning accuracy on Kinetics-400 and Something-Something-v2.
  • Figure 4: Convergence of MAE and ConvMAE on various tasks.