Masking Improves Contrastive Self-Supervised Learning for ConvNets, and Saliency Tells You Where

TL;DR

This paper tackles the challenge of applying masking-based augmentation to contrastive self-supervised learning with convolutional networks. It introduces a saliency-guided masking framework that uses a frozen SCDA-based localization to distribute masked patches across foreground and background, supported by three masking strategies and the creation of hard negatives. Empirically, the method improves performance over MSCN and ADIOS across ImageNet-100 classification and a suite of transfer tasks, and ablation studies validate the importance of saliency guidance, masking only the query branch, and hard negatives. The approach offers a practical, efficient way to leverage masking in ConvNet-based SSL, with broad implications for robust, transferable representations.

Abstract

While image data starts to enjoy the simple-but-effective self-supervised learning scheme built upon masking and self-reconstruction objective thanks to the introduction of tokenization procedure and vision transformer backbone, convolutional neural networks as another important and widely-adopted architecture for image data, though having contrastive-learning techniques to drive the self-supervised learning, still face the difficulty of leveraging such straightforward and general masking operation to benefit their learning process significantly. In this work, we aim to alleviate the burden of including masking operation into the contrastive-learning framework for convolutional neural networks as an extra augmentation method. In addition to the additive but unwanted edges (between masked and unmasked regions) as well as other adverse effects caused by the masking operations for ConvNets, which have been discussed by prior works, we particularly identify the potential problem where for one view in a contrastive sample-pair the randomly-sampled masking regions could be overly concentrated on important/salient objects thus resulting in misleading contrastiveness to the other view. To this end, we propose to explicitly take the saliency constraint into consideration in which the masked regions are more evenly distributed among the foreground and background for realizing the masking-based augmentation. Moreover, we introduce hard negative samples by masking larger regions of salient patches in an input image. Extensive experiments conducted on various datasets, contrastive learning mechanisms, and downstream tasks well verify the efficacy as well as the superior performance of our proposed method with respect to several state-of-the-art baselines.

Figures (3)

• Figure 1: An overview for our proposed method of including saliency-guided masking augmentation into contrastive self-supervised learning, where the backbone of the feature extractor is ConvNets. Firstly, our localization network $f_{\xi}$ produce the saleincy map which is built upon SCDA wei2017selective (cf. Section \ref{['Object localization']}), in which such saliency map helps to separate the foreground objects and background in an image. Given an input image, after conducting standard augmentations along the query and key branches (following the common practice of siamese network) to produce two views, our proposed saliency-guided masking strategies are adopted to produce positive and hard negative samples (please refer to Section \ref{['Masking strategy']} for more details, where in this figure we take the high-pass filtering strategy as an example). The constructed positive and (hard) negative samples are gone through the feature encoder to compute the contrastive objective function $\mathcal{L}_{nce}$ (please refer to our Section \ref{['Overall design']}). Noting that here we base on the SSL framework of MoCov2 chen2020improved to illustrate the computation flow, hence there exists an momentum encoder $f_{\varepsilon}$ in addition to our main learning target, the feature encoder $f_{\theta}$.
• Figure 2: Comparison in terms of modelling among MSCN jing2022masked, ADIOS shi2022adversarial, and our proposed method. Please refer to the last paragraph of Section \ref{['sec:related']} for more detailed descriptions.
• Figure 3: Three masking strategies (cf. Section \ref{['Masking strategy']}).