Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Masked Siamese Networks for Label-Efficient Learning

Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, Nicolas Ballas

TL;DR

Masked Siamese Networks (MSN) present a discriminative, mask-denoising self-supervised pretraining framework for Vision Transformers that avoids input-level reconstruction. By masking patches in one view and aligning its [CLS] representation to an unmasked view through a shared prototype space with target sharpening and me-max regularization, MSN delivers strong label-efficient performance and scalable training. Empirically, MSN achieves state-of-the-art results in 1% ImageNet-1K (75.7% top-1) and competitive linear/fine-tuning performance, while offering significant compute/memory advantages due to masking. The work demonstrates robust transfer, substantial low-shot gains, and detailed ablations highlighting masking strategies and augmentation invariances as key design levers.

Abstract

We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available.

Masked Siamese Networks for Label-Efficient Learning

TL;DR

Masked Siamese Networks (MSN) present a discriminative, mask-denoising self-supervised pretraining framework for Vision Transformers that avoids input-level reconstruction. By masking patches in one view and aligning its [CLS] representation to an unmasked view through a shared prototype space with target sharpening and me-max regularization, MSN delivers strong label-efficient performance and scalable training. Empirically, MSN achieves state-of-the-art results in 1% ImageNet-1K (75.7% top-1) and competitive linear/fine-tuning performance, while offering significant compute/memory advantages due to masking. The work demonstrates robust transfer, substantial low-shot gains, and detailed ablations highlighting masking strategies and augmentation invariances as key design levers.

Abstract

We propose Masked Siamese Networks (MSN), a self-supervised learning framework for learning image representations. Our approach matches the representation of an image view containing randomly masked patches to the representation of the original unmasked image. This self-supervised pre-training strategy is particularly scalable when applied to Vision Transformers since only the unmasked patches are processed by the network. As a result, MSNs improve the scalability of joint-embedding architectures, while producing representations of a high semantic level that perform competitively on low-shot image classification. For instance, on ImageNet-1K, with only 5,000 annotated images, our base MSN model achieves 72.4% top-1 accuracy, and with 1% of ImageNet-1K labels, we achieve 75.7% top-1 accuracy, setting a new state-of-the-art for self-supervised learning on this benchmark. Our code is publicly available.
Paper Structure (44 sections, 1 theorem, 4 equations, 9 figures, 16 tables)

This paper contains 44 sections, 1 theorem, 4 equations, 9 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Problem Formulation
  4. Siamese Networks
  5. Vision Transformer
  6. Masked Siamese Networks
  7. Input Views
  8. Patchify and Mask
  9. Encoder
  10. Similarity Metric and Predictions
  11. Training Objective
  12. Related Work
  13. Results
  14. Label-Efficient Learning
  15. Extreme Low-Shot
  16. ...and 29 more sections

Key Result

Proposition 1

Suppose Assumption ass:sharp holds. If $f_\theta(\cdot)$ is such that the representations collapse, i.e., $z_{i,m} = z_{j,k}$ for all $i,j \in [B]$ and $m,k \in [M]$, then $\left\lVert\nabla_\theta H(p^+_{i}, p_{i,m})\right\rVert + \left\lVert\nabla_\theta H(\overline{p})\right\rVert > 0$ for all $i

Figures (9)

  • Figure 1: Low-shot Evaluation of self-supervised models, pre-trained on ImageNet-1K. (Left) MSN surpasses the previous 800M parameter state-of-the-art, while using a model that is $10\times$ smaller. (Right) MSN achieves good classification performance using less labels than current mask-based auto-encoders.
  • Figure 2: Visualization of MSN representations. First column: original image. Second column: image with 70% of patches masked, input to an MSN pre-trained ViT-L/7 encoder to compute representations. Other columns: Samples of a generative model conditioned on the MSN representations (see Appendix \ref{['apndx:qualitative']} for more details and other samples). Qualities that vary across samples represent information that the pre-trained representation is invariant to; e.g., in this case, MSN discards background, pose, and lighting information. Qualities that are common across samples represent information that the pre-trained representation is not invariant to. In this case, even with a large fraction of the patches corrupted with mask noise, MSN representations still encode semantic information about the object of interest.
  • Figure 3: Masked Siamese Networks. First use random data augmentations to generate two views of an image, referred to as the anchor view and the target view. Subsequently, a random mask is applied to the anchor view, while the target view is left unchanged. The objective is then to assign the representation of the masked anchor view to the same clusters as the representation of the unmasked target view. A standard cross-entropy loss is used as the criterion to optimize.
  • Figure 4: Masking strategies. When applying a Random Mask, we randomly drop patches across a global view of the image. When applying a Focal Mask, we randomly select a local continuous block of an image, and mask everything around it. We typically leverage both Random and Focal Masking strategies when pre-training with MSNs.
  • Figure 5: Visualizations of ViT-B/8 pre-trained representations computed from images with 50% of patches masked. First column: original image. Second column: image with 50% of patches masked used to compute representations of an SSL pre-trained ViT-B/8 encoder. Other columns: RCDM sampling from generative model conditioned on SSL representation of masked image.
  • ...and 4 more figures

Theorems & Definitions (2)

  • Proposition 1: Non-Collapsing Representations
  • proof