HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms

TL;DR

An adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively that can be integrated across a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training.

Abstract

In this paper, we propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. SSL approaches typically work through some invariance term to ensure consistency between similar samples and a regularization term to prevent global dimensional collapse. Dimensional collapse refers to data representations spanning a lower-dimensional subspace. Recent work has demonstrated that the representation space of these algorithms gradually reflects a semantic hierarchical structure as training progresses. Data samples of the same hierarchical grouping tend to exhibit greater dimensional collapse locally compared to the dataset as a whole due to sharing features in common with each other. Ideally, SSL algorithms would take advantage of this hierarchical emergence to have an additional regularization term to account for this local dimensional collapse effect. However, the construction of existing SSL algorithms does not account for this property. To address this, we propose an adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively. This decomposition is based on an adaptive threshold that gradually lowers to reflect the emerging hierarchical structure of the representation space throughout training. It is based on an analysis of the cosine similarity distribution of samples in a batch. We demonstrate that this hierarchical emergence exploitation (HEX) approach can be integrated across a wide variety of SSL algorithms. Empirically, we show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.

Figures (15)

• Figure 1: a) Traditional view of dimensional collapse is all samples mapping to a single point. b) HEX view of dimensional collapse is that samples first collapse to local hierarchical regions before collapsing globally.
• Figure 2: a) Traditional contrastive learning algorithms do not adapt their optimization objective to changes in the structure of the representation space as training progresses. b) HEX introduces a custom negative repulsion to samples above a $\epsilon$ threshold cosine similarity of the anchor image. This $\epsilon$ threshold is varied during training to reflect changes in the structure of the representation space. Early and late training refers to the explicit training time of the algorithm i.e. the number of epochs.
• Figure 3: Effective rank of 500 sample subsets that are sampled from only superclass samples and random samples across the Cifar-100 test set from a model trained with NNCLR alone and NNCLR + HEX. The effective rank was calulated using the RankMe metric.
• Figure 4: a) and b) are plots of the cosine similarity distribution at different epochs of training for the SuperClass samples and Regular samples respectively on the Cifar-100 test set. c) This is a plot of the skew of the cosine similarity distribution for both superclass and regular samples across training.
• Figure 5: This shows the overall process by which the HEX loss is computed. Embeddings are divided by a cosine similarity threshold $\epsilon$ into a hierarchy $H(i)$ and regular subgroup. Hierarchy samples receive additional regularization with the $Q_{hi}$ function and this construction is integrated into the HEX loss.