Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Senthil Purushwalkam, Abhinav Gupta

TL;DR

The paper investigates why contrastive self-supervised learning gains transfer to vision tasks by quantifying invariances encoded in representations. It introduces a formal framework to measure invariances and reveals that occlusion invariance largely arises from aggressive augmentation and object-centric dataset biases, while viewpoint and instance invariances remain underexplored. To address this, it proposes learning from unstructured videos via Frame Temporal Invariance and Region Tracker, integrating temporal and regional transformations into the contrastive objective ${\mathcal L}$ with MOCOv2, and demonstrates stronger viewpoint, illumination, and instance invariances along with better downstream performance. The findings suggest that beyond augmentation tactics, data bias and temporal structure in videos are key levers for robust, transferable representations, offering practical guidance for designing self-supervised objectives and data curricula.

Abstract

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

TL;DR

The paper investigates why contrastive self-supervised learning gains transfer to vision tasks by quantifying invariances encoded in representations. It introduces a formal framework to measure invariances and reveals that occlusion invariance largely arises from aggressive augmentation and object-centric dataset biases, while viewpoint and instance invariances remain underexplored. To address this, it proposes learning from unstructured videos via Frame Temporal Invariance and Region Tracker, integrating temporal and regional transformations into the contrastive objective with MOCOv2, and demonstrates stronger viewpoint, illumination, and instance invariances along with better downstream performance. The findings suggest that beyond augmentation tactics, data bias and temporal structure in videos are key levers for robust, transferable representations, offering practical guidance for designing self-supervised objectives and data curricula.

Abstract

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

Paper Structure

This paper contains 13 sections, 3 equations, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contrastive Representation Learning
  3. Related Work
  4. Demystifying Contrastive SSL
  5. Measuring Invariances
  6. Augmentation and Dataset Biases
  7. Learning from Videos
  8. Evaluating Temporal Invariance Models
  9. Analyzing Temporal Invariance Models
  10. Conclusion
  11. Acknowledgements
  12. Comparison of Invariance Measure to Goodfellow et. alinvariances
  13. Implementation Details: Learning from Videos

Figures (2)

  • Figure 1: Aggressive Augmentation Constrastive self-supervised learning methods employ an aggressive cropping strategy to generate positive pairs. Through this strategy, an image (left) yields many non-overlapping crops (right) as samples. We can observe that the crops do not necessarily depict objects of the same category. Therefore, a representation that matches features of these crops would be detrimental for downstream object recognition tasks.
  • Figure 2: Leveraging Temporal Transformations: We propose an approach to leverage the naturally occurring transformations in videos and learn representations in the MOCOv2 framework. The Frame Temporal Invariance model uses full frames and tracked region proposals separated in time as the query and key. See supplementary material Appendix \ref{['supsec:implementation']} for additional implementation details.