Hypersolid: Emergent Vision Representations via Short-Range Repulsion

TL;DR

This work reframes self-supervised vision learning as a discrete packing problem and introduces Hypersolid, a method that uses short-range repulsion with an exclusion radius $\alpha$ to prevent representation collapse while preserving augmentation diversity. By combining a max-pooled alignment target with a weak $L_2$ normalization, Hypersolid promotes near-injectivity in the embedding space, achieving robust inter-class separation without sacrificing intra-class diversity. Theoretical justification ties the approach to the Data Processing Inequality and probabilistic collision avoidance, while extensive experiments show competitive ImageNet performance and clear gains on fine-grained and low-resolution datasets such as Food-101 and CIFAR-100. The work also reveals a distinct, isotropic-like latent geometry and introduces metrics like Structure Ratio and $d′$ to characterize representation separability, suggesting a shift from volume-maximization to efficient, discrete packing in SSL.

Abstract

A recurring challenge in self-supervised learning is preventing representation collapse. Existing solutions typically rely on global regularization, such as maximizing distances, decorrelating dimensions or enforcing certain distributions. We instead reinterpret representation learning as a discrete packing problem, where preserving information simplifies to maintaining injectivity. We operationalize this in Hypersolid, a method using short-range hard-ball repulsion to prevent local collisions. This constraint results in a high-separation geometric regime that preserves augmentation diversity, excelling on fine-grained and low-resolution classification tasks.

Figures (16)

• Figure 1: Hypersolid Qualitative Feature Analysis (ResNet-50 on ImageNet-1k). Left to Right: Input image, hypercolumn PCA, multi-layer Grad-CAM, and gradient-based feature inversion. Note the emergent semantic segmentation (warm colors on foregrounds), the foreground-oriented focusing bias, and the retention of fine-grained compositional details.
• Figure 2: Hypersolid Training Workflow. An input image is augmented into global and local views and encoded. Top (Purple): Views are aligned to a "Feature Union" target created by max-pooling embeddings (with stop-gradient). Middle (Red): Short-range repulsion penalizes any pair (positive or negative) exceeding similarity $\alpha$. Bottom (Yellow): A weak $L_2$ penalty regularizes feature magnitude.
• Figure 3: Pairwise Cosine Similarity Distributions (ImageNet-1000).
• Figure 4: Semantic Topology Analysis on ImageNet-1000. Potential energy barriers for interpolation paths between random pairs, where solid lines represent the mean energy and shaded regions indicate $\pm 1$ standard deviation across all pairs.
• Figure 5: Qualitative analysis of a ViT-Tiny trained on STL-10. Despite the limited capacity, data regime (STL-10) and smaller batch size, the model attention still bias towards "foreground objects". Due to lower resolution of the ViT-Tiny patch tokens the PCA projection is harder to appreciate. Still some semantic differentiation can be appreciated, such as the pelicans in brown, the two cats in orange or the horses in cyan.