Masked Image Modeling with Denoising Contrast

TL;DR

ConMIM introduces a tokenizer-free masked image modeling framework that treats masked patch prediction as denoising contrastive learning. By constructing a patch-level dynamic dictionary per image and applying a denoising contrastive objective with asymmetric perturbations and momentum-based targets, ConMIM achieves strong self-supervised pre-training for ViTs across classification, segmentation, and detection tasks. The method delivers competitive results on ImageNet-1K with small and large ViT backbones, and scales effectively with uncurated data, while avoiding extra tokenizers. This approach highlights the shared foundations of contrastive learning and masked image modeling and offers a practical, efficient pathway for strong visual representations in real-world settings.

Abstract

Since the development of self-supervised visual representation learning from contrastive learning to masked image modeling (MIM), there is no significant difference in essence, that is, how to design proper pretext tasks for vision dictionary look-up. MIM recently dominates this line of research with state-of-the-art performance on vision Transformers (ViTs), where the core is to enhance the patch-level visual context capturing of the network via denoising auto-encoding mechanism. Rather than tailoring image tokenizers with extra training stages as in previous works, we unleash the great potential of contrastive learning on denoising auto-encoding and introduce a pure MIM method, ConMIM, to produce simple intra-image inter-patch contrastive constraints as the sole learning objectives for masked patch prediction. We further strengthen the denoising mechanism with asymmetric designs, including image perturbations and model progress rates, to improve the network pre-training. ConMIM-pretrained models with various scales achieve competitive results on downstream image classification, semantic segmentation, object detection, and instance segmentation tasks, e.g., on ImageNet-1K classification, we achieve 83.9% top-1 accuracy with ViT-Small and 85.3% with ViT-Base without extra data for pre-training.

Figures (5)

• Figure 1: Conventional contrastive learning methods (e.g., MoCo moco, SimCLR simclr) and masked image modeling methods (e.g., BEiT beit, PeCo peco) both perform the pretext task of vision dictionary look-up, where the superiority of the latter ones lie in the patch-level denoising auto-encoding mechanism to enable fine-grained visual context understanding of vision Transformers vit. We introduce to cast masked image modeling as denoising contrastive learning to avoid the extra training stages of image tokenizer, rendering a flexible, simple and effective pre-training paradigm.
• Figure 2: Our ConMIM performs the masked patch prediction with denoising contrast, coupling with two asymmetric designs to achieve state-of-the-art performance on self-supervised image pre-training. The slowly progressing vision Transformer is a snapshot of the backbone network under training, and we do not require any off-the-shelf image tokenizers. The training objective of denoising contrastive loss performs a patch-level look-up from dynamic vision dictionaries and enhances the network's capability to capture more fine-grained visual context.
• Figure 3: Visualize the self-attention map between [CLS] token and local tokens of the pre-trained ViT-B/16 vit model on ImageNet-1K imagenet, where (a) indicates ConMIM pretraining and (b) indicates the vanilla instance-level contrastive pre-training. Self-attention maps out of 12 attention heads are averaged. It can be observed that ConMIM-pretrained models are much more locally discriminative and aware of the visual context.
• Figure 4: Visualize the dynamic dictionary composed of patches. The dictionary in ConMIM properly provides positive keys with similar semantics while the baseline tokenizer is vulnerable to various low-level changes. (a) The query patch from ImageNet validation set. (b) Top-ranked patches retrieved by ConMIM-pretrained model. (c) Patches out of the same ID (#1813) tokenized by dVAE dalle in baseline BEiT beit.
• Figure 5: Partial fine-tuning of ConMIM-pretrained ViT-B/16 model.