Efficient Masked Autoencoders with Self-Consistency

TL;DR

EMAE tackles two major drawbacks of high-ratio masked image modeling: inefficient data utilization and inconsistent reconstructions. It introduces a parallel mask strategy to achieve near-complete data utilization and a self-consistency loss to stabilize predictions across overlapping patches. Together, these components yield substantially faster pre-training and improved transfer to image classification, object detection, and semantic segmentation, with state-of-the-art results on several benchmarks. The approach demonstrates strong efficiency and robustness across diverse datasets (ImageNet, COCO, OpenImages) and ViT backbones, offering a practical path toward reliable, self-supervised representations for dense and generic Vision tasks.

Abstract

Inspired by the masked language modeling (MLM) in natural language processing tasks, the masked image modeling (MIM) has been recognized as a strong self-supervised pre-training method in computer vision. However, the high random mask ratio of MIM results in two serious problems: 1) the inadequate data utilization of images within each iteration brings prolonged pre-training, and 2) the high inconsistency of predictions results in unreliable generations, $i.e.$, the prediction of the identical patch may be inconsistent in different mask rounds, leading to divergent semantics in the ultimately generated outcomes. To tackle these problems, we propose the efficient masked autoencoders with self-consistency (EMAE) to improve the pre-training efficiency and increase the consistency of MIM. In particular, we present a parallel mask strategy that divides the image into K non-overlapping parts, each of which is generated by a random mask with the same mask ratio. Then the MIM task is conducted parallelly on all parts in an iteration and the model minimizes the loss between the predictions and the masked patches. Besides, we design the self-consistency learning to further maintain the consistency of predictions of overlapping masked patches among parts. Overall, our method is able to exploit the data more efficiently and obtains reliable representations. Experiments on ImageNet show that EMAE achieves the best performance on ViT-Large with only 13% of MAE pre-training time using NVIDIA A100 GPUs. After pre-training on diverse datasets, EMAE consistently obtains state-of-the-art transfer ability on a variety of downstream tasks, such as image classification, object detection, and semantic segmentation.

Figures (4)

• Figure 1: Different reconstruction results of MAE he2021masked correspond to different mask seeds. (a) Different combinations sampled by different mask seeds. (b) Reconstruction results of MAE. For the three reconstructions of (b), only the first represents a normal cattle, and the third even reconstructs a dog. The semantics of these reconstructions by MAE are inconsistent.
• Figure 2: Illustration of our EMAE. The whole image is first divided into $N$ patches, then the parallel strategy divides $N$ patches into $K$ non-overlapping parts $\mathbf{x}_{v_1}$, $...$, $\mathbf{x}_{v_K}$ of the same size. Each part consists of $N/K$ random non-overlapping visible image patches. Then, each part is fed into the encoder-decoder architecture and performs the MIM task to generate $\mathbf{x}_{m_1}$, $...$, and $\mathbf{x}_{m_K}$. Furthermore, self-consistency learning guides the overlapping predictions in identical positions to be pulled together. Here, we take $\mathbf{x}_{p_1}$ and $\mathbf{x}_{p_K}$ as examples.
• Figure 3: Different predictions of identical positions of MIM pre-trained model using different mask seeds. P$_{1}$ and P$_{2}$ are the predictions of identical positions, and gt is the ground truth. Both P$_{1}$ and P$_{2}$ are similar to gt, however, the semantics between P$_{1}$ and P$_{2}$ are dissimilar.
• Figure 4: Different reconstruction results of our EMAE correspond to different mask seeds. Different combinations of visible patches are sampled from the same image by the random seeds, then these combinations are fed into our EMAE and EMAE generates the reconstructed images. Here, we reconstruct three images as examples. These reconstruction results contain similar semantics and become matched closely with each other, demonstrating the effectiveness of our self-consistency learning.