Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?

TL;DR

The surprising observation that the additional designs introduced by SSL do not contribute to the quality of the learned representations is brought forward, which provides legitimacy to existing theoretical studies, but also simplifies the practitioner's path to SSL deployment in numerous small and medium scale settings.

Abstract

Deep Learning is often depicted as a trio of data-architecture-loss. Yet, recent Self Supervised Learning (SSL) solutions have introduced numerous additional design choices, e.g., a projector network, positive views, or teacher-student networks. These additions pose two challenges. First, they limit the impact of theoretical studies that often fail to incorporate all those intertwined designs. Second, they slow-down the deployment of SSL methods to new domains as numerous hyper-parameters need to be carefully tuned. In this study, we bring forward the surprising observation that--at least for pretraining datasets of up to a few hundred thousands samples--the additional designs introduced by SSL do not contribute to the quality of the learned representations. That finding not only provides legitimacy to existing theoretical studies, but also simplifies the practitioner's path to SSL deployment in numerous small and medium scale settings. Our finding answers a long-lasting question: the often-experienced sensitivity to training settings and hyper-parameters encountered in SSL come from their design, rather than the absence of supervised guidance.

Figures (10)

• Figure 1: DIET's training loss is indicative of downstream test performance. We depict DIET's training loss ( y-axis) against the online test linear probe accuracy ( x-axis) for all the models, hyper-parameters, and training epochs. Yellow to purple correspond to different label smoothing which plays a role in DIET's convergence speed (\ref{['sec:ablation']}). For a given label smoothing parameter, there exists a strong relationship between DIET's training loss and the downstream test accuracy enabling label-free quantitative quality assessment one's model.
• Figure 2: DIET matches supervised learning on datasets with only a few samples per class. Depiction of DIET's downstream performances ( blue) against supervised learning ( red) controlling training set size ( x-axis); evaluation is performed over the original full evaluation set. DIET is able to learn highly competitive representations when the dataset is small with only a few samples per classes. See \ref{['fig:ablation']} for additional datasets.
• Figure 3: DIET uses the datum index (n) as the class-target --effectively turning unsupervised learning into a supervised learning problem. In our case, we employ the cross-entropy loss (X-Ent), no extra care needed to handle different dataset or architectures. As opposed to current SOTA, we do not rely on a projector nor positive views i.e no change needs to be done to any existing supervised pipeline to obtain DIET. As highlighted in \ref{['fig:accus']}, DIET's training loss is even informative of downstream test performances, and as ablated in \ref{['sec:ablation']} there is no degradation of performance with longer training, even for very small datasets (\ref{['tab:transfer']}).
• Figure 4: Empirical validation of \ref{['sec:linear_model']} depicting the optimal solution for DIET for the parameters ${\bm{W}}$ and ${\bm{V}}$ under a clustered input data assumption ( left column), in this case, made of four clusters with four samples per cluster. The learned ${\bm{W}}$ given in the middle column converge to the same clustering, as predicted by our closed-form solution. We also obtain in the right column the evolution of the DIET training loss that we compare against the optimal value of the loss (obtained from the optimal parameters). We see that the training converges towards the optimal value of the loss (up to 1e-7 at the end of that training episode).
• Figure 5: Depiction of the optimal ${\bm{A}}$ matrix (recall \ref{['eq:A']}) on the right, obtained empirically from inserting the optimal parameters that we found for ${\bm{W}}$ and ${\bm{V}}$. As predicted by \ref{['eq:A']} that matrix is made of blocks aligned with the original clustering of the input data matrix ${\bm{X}}$ given on the left.