Semi-Supervised Contrastive Learning with Orthonormal Prototypes
Huanran Li, Manh Nguyen, Daniel Pimentel-Alarcón
TL;DR
The paper addresses dimensional collapse in semi-supervised contrastive learning by identifying a critical learning-rate threshold that drives collapse under InfoNCE. It introduces CLOP, a loss that couples standard contrastive learning with a supervised term guiding embeddings toward orthonormal class prototypes, thereby increasing embedding rank and separability. Through extensive classification and transfer-learning experiments, CLOP demonstrates superior performance and stability across learning rates and batch sizes, with robust ablations validating the role of orthonormal prototypes. The work offers a practical approach to stabilize SSL representations and extend effective semi-supervised learning to vision tasks, while noting limitations in fixed prototype counts and initialization.
Abstract
Contrastive learning has emerged as a powerful method in deep learning, excelling at learning effective representations through contrasting samples from different distributions. However, dimensional collapse, where embeddings converge into a lower-dimensional space, poses a significant challenge, especially in semi-supervised and self-supervised setups. In this paper, we first identify a critical learning-rate threshold, beyond which standard contrastive losses converge to collapsed solutions. Building on these insights, we propose CLOP, a novel semi-supervised loss function designed to prevent dimensional collapse by promoting the formation of orthogonal linear subspaces among class embeddings. Through extensive experiments on real and synthetic datasets, we demonstrate that CLOP improves performance in image classification and object detection tasks while also exhibiting greater stability across different learning rates and batch sizes.