Label-free Monitoring of Self-Supervised Learning Progress

TL;DR

This work addresses the challenge of evaluating SSL encoder quality without labeled data. It introduces label-free evaluation metrics based on $k$-means clustering (with $S_1$ and $AMI(C_1; C_2)$) and embedding entropy $H(Z_3)$, benchmarking them against linear-probe accuracy across SimSiam, SimCLR, and MoCo-v2 on CIFAR-10/100 and ImageNet-pretrained encoders. The results show that $AMI(C_1; C_{GT})$ correlates strongly with LP for SimCLR and MoCo-v2, while SimSiam presents weaker, more variable signals; entropy exhibits mixed, method-dependent behavior and can be architecture-dependent in some settings. Clustering-based metrics struggle to generalize across architectures, while entropy shows potential as an architecture-independent signal in specific contexts, underscoring the need for broader evaluation across methods. Overall, the proposed label-free metrics offer a practical monitor for SSL progress in certain setups but require further study to generalize across architectures and SSL paradigms.

Abstract

Self-supervised learning (SSL) is an effective method for exploiting unlabelled data to learn a high-level embedding space that can be used for various downstream tasks. However, existing methods to monitor the quality of the encoder -- either during training for one model or to compare several trained models -- still rely on access to annotated data. When SSL methodologies are applied to new data domains, a sufficiently large labelled dataset may not always be available. In this study, we propose several evaluation metrics which can be applied on the embeddings of unlabelled data and investigate their viability by comparing them to linear probe accuracy (a common metric which utilizes an annotated dataset). In particular, we apply $k$-means clustering and measure the clustering quality with the silhouette score and clustering agreement. We also measure the entropy of the embedding distribution. We find that while the clusters did correspond better to the ground truth annotations as training of the network progressed, label-free clustering metrics correlated with the linear probe accuracy only when training with SSL methods SimCLR and MoCo-v2, but not with SimSiam. Additionally, although entropy did not always have strong correlations with LP accuracy, this appears to be due to instability arising from early training, with the metric stabilizing and becoming more reliable at later stages of learning. Furthermore, while entropy generally decreases as learning progresses, this trend reverses for SimSiam. More research is required to establish the cause for this unexpected behaviour. Lastly, we find that while clustering based approaches are likely only viable for same-architecture comparisons, entropy may be architecture-independent.

Figures (6)

• Figure 1: Overview of SSL methodologies. For a given image, $x$, two views of the image ($x_i$ and $x_j$) are created with randomly sampled augmentations. In SimSiam (\ref{['fig:simsiam-arch']}) and SimCLR (\ref{['fig:simclr-arch']}), the two views are passed through the same encoder, $f$, and projector, $g$, layers; in MoCo-v2 (\ref{['fig:mocov2-arch']}), view $x_j$ passes through a moving average model instead. For SimSiam, the loss is the similarity between the output of a predictor head on the $x_i$ branch and the projection of $x_j$; whereas for SimCLR and MoCo-v2, InfoNCE is used with negative samples drawn from the batch (SimCLR) or the memory queue (MoCo). In SimSiam and MoCo-v2, a stop-gradient is applied to prevent the loss returning down the $x_j$ branch of the network.
• Figure 2: Entropy progression w.r.t. training for SimCLR on CIFAR-100. We highlight milestones with outlying samples in the embedding space (orange).
• Figure 3: Entropy progression w.r.t training for MoCo-v2 on CIFAR-100.
• Figure 4: Entropy progression w.r.t training for SimSiam on CIFAR-100.
• Figure 5: Entropy vs linear probe on CIFAR-10, for networks pre-trained on ImageNet.