Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond

Chaoning Zhang, Chenshuang Zhang, Junha Song, John Seon Keun Yi, Kang Zhang, In So Kweon

TL;DR

The paper surveys masked autoencoders as a scalable self-supervised learning framework for vision, tracing its revival from NLP-inspired masked modeling (e.g., BERT) to end-to-end image pretraining (MAE) and beyond. It analyzes BEiT and MAE variants, efficiency improvements, theoretical perspectives, and the interplay with joint-embedding methods, showing MAE’s strong downstream transfer and flexibility across architectures. The survey further expands MAE’s reach into video, medical imaging, vision-language tasks, and non-visual domains like point clouds, graphs, RL, and audio, illustrating the method’s broad applicability and potential. Collectively, the work provides a structured understanding of why masked autoencoding works, how to design efficient and effective pretraining, and where the approach can impact future multi-modal and cross-domain SSL research.

Abstract

Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. By contrast, early attempts at generative methods in vision have been buried by their discriminative counterparts (like contrastive learning); however, the success of mask image modeling has revived the masking autoencoder (often termed denoising autoencoder in the past). As a milestone to bridge the gap with BERT in NLP, masked autoencoder has attracted unprecedented attention for SSL in vision and beyond. This work conducts a comprehensive survey of masked autoencoders to shed insight on a promising direction of SSL. As the first to review SSL with masked autoencoders, this work focuses on its application in vision by discussing its historical developments, recent progress, and implications for diverse applications.

A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond

TL;DR

The paper surveys masked autoencoders as a scalable self-supervised learning framework for vision, tracing its revival from NLP-inspired masked modeling (e.g., BERT) to end-to-end image pretraining (MAE) and beyond. It analyzes BEiT and MAE variants, efficiency improvements, theoretical perspectives, and the interplay with joint-embedding methods, showing MAE’s strong downstream transfer and flexibility across architectures. The survey further expands MAE’s reach into video, medical imaging, vision-language tasks, and non-visual domains like point clouds, graphs, RL, and audio, illustrating the method’s broad applicability and potential. Collectively, the work provides a structured understanding of why masked autoencoding works, how to design efficient and effective pretraining, and where the approach can impact future multi-modal and cross-domain SSL research.

Abstract

Masked autoencoders are scalable vision learners, as the title of MAE \cite{he2022masked}, which suggests that self-supervised learning (SSL) in vision might undertake a similar trajectory as in NLP. Specifically, generative pretext tasks with the masked prediction (e.g., BERT) have become a de facto standard SSL practice in NLP. By contrast, early attempts at generative methods in vision have been buried by their discriminative counterparts (like contrastive learning); however, the success of mask image modeling has revived the masking autoencoder (often termed denoising autoencoder in the past). As a milestone to bridge the gap with BERT in NLP, masked autoencoder has attracted unprecedented attention for SSL in vision and beyond. This work conducts a comprehensive survey of masked autoencoders to shed insight on a promising direction of SSL. As the first to review SSL with masked autoencoders, this work focuses on its application in vision by discussing its historical developments, recent progress, and implications for diverse applications.
Paper Structure (27 sections, 4 figures, 4 tables)

This paper contains 27 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and terminology
  3. Masked autoencoding: NLP to vision
  4. NLP and vision followed different SSL paths
  5. Is generative SSL suitable for vision?
  6. Summary and remark
  7. Masked autoencoder for image modeling
  8. BEiT and its improved variants
  9. End-to-end masked autoencoder
  10. Towards improving efficiency
  11. Various perspectives on the success of masked autoencoder in vision
  12. Relationship with joint-embedding methods
  13. Masked autoencoder and joint embedding: boosting each other
  14. Masked autoencoder and joint embedding: bridging their gap
  15. Other Applications: Vision and Beyond
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Timeline of Visual SSL
  • Figure 2: Overview of BEIT pre-training. The figure is edited from bao2022beit.
  • Figure 3: Overview of a masked autoencoder with the figure borrowed from the original work MAE he2022masked.
  • Figure 4: Various masking strategies in SimMIM with the figure borrowed from the original paper xie2022simmim.