Enhancing Self-Supervised Learning with Semantic Pairs A New Dataset and Empirical Study

TL;DR

The paper addresses the limited generalization of instance-discrimination SSL caused by reliance on handcrafted data transformations. It introduces a manually curated Semantic Pairs Dataset to provide semantically related image pairs as supervision and compares it against an augmented-pairs baseline across multiple SSL methods. Through transfer learning, object detection, and ablation studies, the work demonstrates that semantic pairs yield substantially better generalization and robustness across architectures and tasks, while reducing dependence on specific augmentations. The findings suggest semantic-aware supervision can enhance the practicality and transferability of self-supervised vision models, with implications for resource-efficient SSL and broader downstream applicability.

Abstract

Instance discrimination is a self-supervised representation learning paradigm wherein individual instances within a dataset are treated as distinct classes. This is typically achieved by generating two disparate views of each instance by applying stochastic transformations, encouraging the model to learn representations invariant to the common underlying object across these views. While this approach facilitates the acquisition of invariant representations for dataset instances under various handcrafted transformations (e.g., random cropping, colour jittering), an exclusive reliance on such data transformations for achieving invariance may inherently limit the model's generalizability to unseen datasets and diverse downstream tasks. The inherent limitation stems from the fact that the finite set of transformations within the data processing pipeline is unable to encompass the full spectrum of potential data variations. In this study, we provide the technical foundation for leveraging semantic pairs to enhance the generalizability of the model's representation and empirically demonstrate that incorporating semantic pairs mitigates the issue of limited transformation coverage. Specifically, we propose that by exposing the model to semantic pairs (i.e., two instances belonging to the same semantic category), we introduce varied real-world scene contexts, thereby fostering the development of more generalizable object representations. To validate this hypothesis, we constructed and released a novel dataset comprising curated semantic pairs and conducted extensive experimentation to empirically establish that their inclusion enables the model to learn more general representations, ultimately leading to improved performance across diverse downstream tasks.

Figures (7)

• Figure 1: This diagram illustrates how the instance discrimination approaches treat (a) augmented pairs (i.e., two views for the same instance) and (b) semantic pairs (i.e., two instances belonging to the same category) during representation learning.
• Figure 2: Examples from the semantic pairs dataset illustrate placing objects in various real-world context scenes.
• Figure 3: This diagram provides an overview of the four-stage framework for fair model comparison between models trained on augmented pairs (baseline) versus semantic data pairs.
• Figure 4: Transfer learning performance of SOTA approaches pre-trained on Semantic Pairs (SP, striped) and Augmented Pairs (AP, solid), evaluated on CIFAR10, CIFAR100, and STL10.(SP) pre-training consistently improves downstream accuracy over (AP). ($\Delta$) values denote SP-AP differences.
• Figure 5: Performance comparison of SimCLR (SP) and SimCLR (AP) pre-trained for 200 epochs with transformation ablations: (1) Gray$\_$s (grayscale removed), (2) color$\_$jitter (both color jitter and grayscale removed), and (3) only$\_$c (only random crop retained).