Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data

Atik Faysal, Mohammad Rostami, Reihaneh Gh. Roshan, Nikhil Muralidhar, Huaxia Wang

TL;DR

SSMAE tackles data efficiency for Vision Transformers by integrating masked autoencoding with semi-supervised classification through a dynamic pseudo-labeling gate. By jointly optimizing reconstruction on all data and supervised signals from labeled samples, and by admitting pseudo-labels only when predictions are high-confidence and consistent across weak/strong augmentations, it mitigates confirmation bias. Experimental results on CIFAR-10 and CIFAR-100 show meaningful gains in low-label regimes, with CIFAR-10 reaching 56.80% accuracy at 10% labels and CIFAR-100 reaching 22.65% at 10% labels, outperforming strong baselines. Ablation studies confirm the critical roles of reconstruction, consistency regularization, and the gating mechanism, highlighting that when pseudo-labels are introduced matters as much as how they are generated. Overall, SSMAE demonstrates robust, data-efficient transformer pretraining by fusing self-supervised representations with reliable pseudo-label supervision.

Abstract

We address the challenge of training Vision Transformers (ViTs) when labeled data is scarce but unlabeled data is abundant. We propose Semi-Supervised Masked Autoencoder (SSMAE), a framework that jointly optimizes masked image reconstruction and classification using both unlabeled and labeled samples with dynamically selected pseudo-labels. SSMAE introduces a validation-driven gating mechanism that activates pseudo-labeling only after the model achieves reliable, high-confidence predictions that are consistent across both weakly and strongly augmented views of the same image, reducing confirmation bias. On CIFAR-10 and CIFAR-100, SSMAE consistently outperforms supervised ViT and fine-tuned MAE, with the largest gains in low-label regimes (+9.24% over ViT on CIFAR-10 with 10% labels). Our results demonstrate that when pseudo-labels are introduced is as important as how they are generated for data-efficient transformer training. Codes are available at https://github.com/atik666/ssmae.

Semi-Supervised Masked Autoencoders: Unlocking Vision Transformer Potential with Limited Data

TL;DR

SSMAE tackles data efficiency for Vision Transformers by integrating masked autoencoding with semi-supervised classification through a dynamic pseudo-labeling gate. By jointly optimizing reconstruction on all data and supervised signals from labeled samples, and by admitting pseudo-labels only when predictions are high-confidence and consistent across weak/strong augmentations, it mitigates confirmation bias. Experimental results on CIFAR-10 and CIFAR-100 show meaningful gains in low-label regimes, with CIFAR-10 reaching 56.80% accuracy at 10% labels and CIFAR-100 reaching 22.65% at 10% labels, outperforming strong baselines. Ablation studies confirm the critical roles of reconstruction, consistency regularization, and the gating mechanism, highlighting that when pseudo-labels are introduced matters as much as how they are generated. Overall, SSMAE demonstrates robust, data-efficient transformer pretraining by fusing self-supervised representations with reliable pseudo-label supervision.

Abstract

We address the challenge of training Vision Transformers (ViTs) when labeled data is scarce but unlabeled data is abundant. We propose Semi-Supervised Masked Autoencoder (SSMAE), a framework that jointly optimizes masked image reconstruction and classification using both unlabeled and labeled samples with dynamically selected pseudo-labels. SSMAE introduces a validation-driven gating mechanism that activates pseudo-labeling only after the model achieves reliable, high-confidence predictions that are consistent across both weakly and strongly augmented views of the same image, reducing confirmation bias. On CIFAR-10 and CIFAR-100, SSMAE consistently outperforms supervised ViT and fine-tuned MAE, with the largest gains in low-label regimes (+9.24% over ViT on CIFAR-10 with 10% labels). Our results demonstrate that when pseudo-labels are introduced is as important as how they are generated for data-efficient transformer training. Codes are available at https://github.com/atik666/ssmae.
Paper Structure (22 sections, 18 equations, 3 figures, 3 tables)

This paper contains 22 sections, 18 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semi-supervised Learning
  4. Masking-based Representation Learning
  5. Pseudo-labeling
  6. Methodology: SSMAE
  7. Masking Strategy and Encoder Architecture
  8. Decoder Architecture and Reconstruction
  9. Classification Head
  10. Dual-Mode Training Objective
  11. Dynamic Gating Mechanism
  12. Experimental Setup
  13. Datasets
  14. Baselines
  15. Implementation Details
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Overview of the SSMAE framework. A shared encoder is trained on two tasks: masked image reconstruction for all data, and classification for labeled data. For unlabeled data, our dynamic gate generates high-confidence pseudo-labels, which are then included in supervised classification.
  • Figure 2: Reconstruction results on CIFAR-10 and CIFAR-100 datasets. Each visualization shows three columns repreated once: masked input (left), SSMAE reconstruction (center), and original image (right).
  • Figure 3: Effect of different masking ratios.