Do Self-Supervised and Supervised Methods Learn Similar Visual Representations?

TL;DR

The study investigates whether self-supervised and supervised visual representations converge by comparing SimCLR to supervised learning using CK A on CIFAR-10 with a ResNet-50 backbone. It finds that intermediate representations are surprisingly similar despite different objectives, while final-layer representations diverge as each method optimizes for its own goals; notably, SimCLR learns augmentation invariance while supervised emphasizes class mapping, and the similarity of intermediate features largely explains downstream performance. The work demonstrates the pivotal role of intermediate representations and prompts questions about auxiliary task design to encourage shared features. Overall, it suggests that achieving similarity in final representations is not necessary for strong performance, and that aligning intermediate features may be a more fruitful direction for cross-method robustness and transfer.

Abstract

Despite the success of a number of recent techniques for visual self-supervised deep learning, there has been limited investigation into the representations that are ultimately learned. By leveraging recent advances in the comparison of neural representations, we explore in this direction by comparing a contrastive self-supervised algorithm to supervision for simple image data in a common architecture. We find that the methods learn similar intermediate representations through dissimilar means, and that the representations diverge rapidly in the final few layers. We investigate this divergence, finding that these layers strongly fit to their distinct learning objectives. We also find that the contrastive objective implicitly fits the supervised objective in intermediate layers, but that the reverse is not true. Our work particularly highlights the importance of the learned intermediate representations, and raises critical questions for auxiliary task design.

Figures (5)

• Figure 1: CKA between layers of R50 networks trained via SimCLR. We show all, odd, and even layers in the left, middle, and right plots respectively. In contrast to prior work, we compare across different initializations as a sanity check for solution stability.
• Figure 2: (Left/Middle) CKA between the odd/even layers of networks trained by SimCLR and . For the evens, we mark the most similar layer for each SimCLR layer with a white dot. (Right) For each even layer in SimCLR, the similarity to its corresponding supervised layer (diag), and to the most similar supervised layer (max). We also denote the (see \ref{['app:even-odd']}).
• Figure 3: (Left) Linear probe accuracies for learned representations in the SimCLR and models. (Middle) CKA between representations of differently augmented datasets at corresponding layers. (Right) CKA of learned representations with the class representations. We plot post-residual (even) layers only, denoting the block groups (BG) and NCE head (Head) where appropriate.
• Figure 4: between all layers of ResNet-50 networks trained via supervision. We plot all layers in the left column, and even/odd layers on the middle/right.
• Figure 5: We apply the method-specific training augmentations to the CIFAR-10 test dataset and plot the CKA of the representations as they propagate through the supervised and SimCLR models.