Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Bootstrapped Masked Autoencoders for Vision BERT Pretraining

Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, Nenghai Yu

TL;DR

BootMAE advances vision BERT-style pretraining by introducing a momentum encoder to provide evolving target representations and a target-aware decoder that decouples target-specific information from the encoder. The architecture comprises an encoder for structural semantics, a pixel-level regressor, a high-level feature predictor, and cross-attention–based feature injection to supply context to decoders. Empirical results show BootMAE improves ImageNet-1K accuracy over MAE (ViT-B: ~84.2%; ViT-L: ~85.9%) and yields downstream gains on ADE20K and COCO, while enabling more efficient pretraining. Ablation studies validate the benefits of bootstrapped feature prediction, target-aware decoding, and masking strategy choices, highlighting the encoder’s focus on semantic modeling.

Abstract

We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining. The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance. Therefore, we add a momentum encoder in parallel with the original MAE encoder, which bootstraps the pretraining performance by using its own representation as the BERT prediction target. In the second design, we introduce target-specific information (e.g., pixel values of unmasked patches) from the encoder directly to the decoder to reduce the pressure on the encoder of memorizing the target-specific information. Thus, the encoder focuses on semantic modeling, which is the goal of BERT pretraining, and does not need to waste its capacity in memorizing the information of unmasked tokens related to the prediction target. Through extensive experiments, our BootMAE achieves $84.2\%$ Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming MAE by $+0.8\%$ under the same pre-training epochs. BootMAE also gets $+1.0$ mIoU improvements on semantic segmentation on ADE20K and $+1.3$ box AP, $+1.4$ mask AP improvement on object detection and segmentation on COCO dataset. Code is released at https://github.com/LightDXY/BootMAE.

Bootstrapped Masked Autoencoders for Vision BERT Pretraining

TL;DR

BootMAE advances vision BERT-style pretraining by introducing a momentum encoder to provide evolving target representations and a target-aware decoder that decouples target-specific information from the encoder. The architecture comprises an encoder for structural semantics, a pixel-level regressor, a high-level feature predictor, and cross-attention–based feature injection to supply context to decoders. Empirical results show BootMAE improves ImageNet-1K accuracy over MAE (ViT-B: ~84.2%; ViT-L: ~85.9%) and yields downstream gains on ADE20K and COCO, while enabling more efficient pretraining. Ablation studies validate the benefits of bootstrapped feature prediction, target-aware decoding, and masking strategy choices, highlighting the encoder’s focus on semantic modeling.

Abstract

We propose bootstrapped masked autoencoders (BootMAE), a new approach for vision BERT pretraining. BootMAE improves the original masked autoencoders (MAE) with two core designs: 1) momentum encoder that provides online feature as extra BERT prediction targets; 2) target-aware decoder that tries to reduce the pressure on the encoder to memorize target-specific information in BERT pretraining. The first design is motivated by the observation that using a pretrained MAE to extract the features as the BERT prediction target for masked tokens can achieve better pretraining performance. Therefore, we add a momentum encoder in parallel with the original MAE encoder, which bootstraps the pretraining performance by using its own representation as the BERT prediction target. In the second design, we introduce target-specific information (e.g., pixel values of unmasked patches) from the encoder directly to the decoder to reduce the pressure on the encoder of memorizing the target-specific information. Thus, the encoder focuses on semantic modeling, which is the goal of BERT pretraining, and does not need to waste its capacity in memorizing the information of unmasked tokens related to the prediction target. Through extensive experiments, our BootMAE achieves Top-1 accuracy on ImageNet-1K with ViT-B backbone, outperforming MAE by under the same pre-training epochs. BootMAE also gets mIoU improvements on semantic segmentation on ADE20K and box AP, mask AP improvement on object detection and segmentation on COCO dataset. Code is released at https://github.com/LightDXY/BootMAE.
Paper Structure (18 sections, 8 equations, 3 figures, 9 tables)

This paper contains 18 sections, 8 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Approach
  4. Encoder
  5. Feature injection module
  6. Regressor
  7. Predictor
  8. Objective function
  9. Experiments
  10. Implementations
  11. Analysis of BootMAE
  12. ImageNet Classification Comparison
  13. Downstream Tasks
  14. Discussion and Conclusion
  15. Acknowledgement.
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Illustrating the details of our BootMAE in (a) the overall framework and training pipeline, (b) the image encoder block, (c) the feature predictor block & pixel regressor block.
  • Figure 2: Visualization of the two different masking strategies. The masked region is close to the visible region in random masking. For block-wise masking, a large continuous block is masked and most center patches are masked.
  • Figure 3: Performance comparison with MAE in different pre-training epochs using ViT-B and ViT-L, showing that our BootMAE consistently outperforms MAE.