$\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs
Vlad Sobal, Mark Ibrahim, Randall Balestriero, Vivien Cabannes, Diane Bouchacourt, Pietro Astolfi, Kyunghyun Cho, Yann LeCun
TL;DR
X- Sample Contrastive reframes contrastive learning as learning over a soft cross-sample similarity graph, addressing the binary positive/negative limitation by encoding cross-sample relations via a soft adjacency built from captions or class descriptions. The proposed L_{X-CLR} objective, using soft targets derived from a text encoder, yields stronger and more data-efficient representations across ImageNet-scale data and large caption collections, including a 0.6% improvement over CLIP on CC12M for ImageNet and ImageNet Real, and notable gains in background disambiguation. The approach demonstrates robustness to data quality, with label quality playing a critical role in fine-grained attribute Disambiguation, and remains computationally economical by offline similarity precomputation. Together, these results suggest that enriching contrastive objectives with cross-sample semantic signals can produce richer, more generalizable foundation-model representations and can be leveraged for fine-tuning pretrained backbones with limited overhead.
Abstract
Learning good representations involves capturing the diverse ways in which data samples relate. Contrastive loss - an objective matching related samples - underlies methods from self-supervised to multimodal learning. Contrastive losses, however, can be viewed more broadly as modifying a similarity graph to indicate how samples should relate in the embedding space. This view reveals a shortcoming in contrastive learning: the similarity graph is binary, as only one sample is the related positive sample. Crucially, similarities \textit{across} samples are ignored. Based on this observation, we revise the standard contrastive loss to explicitly encode how a sample relates to others. We experiment with this new objective, called $\mathbb{X}$-Sample Contrastive, to train vision models based on similarities in class or text caption descriptions. Our study spans three scales: ImageNet-1k with 1 million, CC3M with 3 million, and CC12M with 12 million samples. The representations learned via our objective outperform both contrastive self-supervised and vision-language models trained on the same data across a range of tasks. When training on CC12M, we outperform CLIP by $0.6\%$ on both ImageNet and ImageNet Real. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8\%$ on ImageNet and $18.1\%$ on ImageNet Real when training with CC3M. Finally, our objective seems to encourage the model to learn representations that separate objects from their attributes and backgrounds, with gains of $3.3$-$5.6$\% over CLIP on ImageNet9. We hope the proposed solution takes a small step towards developing richer learning objectives for understanding sample relations in foundation models.