Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning

Siladittya Manna, Umapada Pal, Saumik Bhattacharya

TL;DR

This paper introduces a binary-information perspective on self-supervised contrastive learning by formulating pretraining as a binary classification over pairwise samples. It develops a sequence of losses—MIOv1, MIOv2 (removing positive–positive repulsion), and MIOv3 (adding an upper bound on negative-pair repulsion)—to maximize mutual information for positive pairs while controlling negative-pair interactions. The authors provide analytic links between the proposed loss and mutual information, derive gradient and Hessian expressions, and establish Lipschitz continuity of the gradient under reasonable assumptions, along with a local PL-based convergence framework for nonconvex SSL. Empirically, MIOv3 achieves state-of-the-art performance on small-scale datasets (CIFAR-10/100, STL-10, Tiny ImageNet) and strong linear-evaluation results on ImageNet100/1K, outperforming several contrastive and non-contrastive methods, with ablations guiding the influence of temperature, training duration, batch size, and parameter count. The work also presents transfer-learning results on medical imaging and provides extensive eigenvalue analyses illustrating optimization dynamics and saddle-point behavior in SSL settings.

Abstract

Self-supervised contrastive learning frameworks have progressed rapidly over the last few years. In this paper, we propose a novel loss function for contrastive learning. We model our pre-training task as a binary classification problem to induce an implicit contrastive effect. We further improve the näive loss function after removing the effect of the positive-positive repulsion and incorporating the upper bound of the negative pair repulsion. Unlike existing methods, the proposed loss function optimizes the mutual information in positive and negative pairs. We also present a closed-form expression for the parameter gradient flow and compare the behaviour of self-supervised contrastive frameworks using Hessian eigenspectrum to analytically study their convergence. The proposed method outperforms SOTA self-supervised contrastive frameworks on benchmark datasets such as CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet. After 200 pretraining epochs with ResNet-18 as the backbone, the proposed model achieves an accuracy of 86.36%, 58.18%, 80.50%, and 30.87% on the CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet datasets, respectively, and surpasses the SOTA contrastive baseline by 1.93%, 3.57%, 4.85%, and 0.33%, respectively. The proposed framework also achieves a state-of-the-art accuracy of 78.4% (200 epochs) and 65.22% (100 epochs) Top-1 Linear Evaluation accuracy on ImageNet100 and ImageNet1K datasets, respectively.

MIO : Mutual Information Optimization using Self-Supervised Binary Contrastive Learning

TL;DR

This paper introduces a binary-information perspective on self-supervised contrastive learning by formulating pretraining as a binary classification over pairwise samples. It develops a sequence of losses—MIOv1, MIOv2 (removing positive–positive repulsion), and MIOv3 (adding an upper bound on negative-pair repulsion)—to maximize mutual information for positive pairs while controlling negative-pair interactions. The authors provide analytic links between the proposed loss and mutual information, derive gradient and Hessian expressions, and establish Lipschitz continuity of the gradient under reasonable assumptions, along with a local PL-based convergence framework for nonconvex SSL. Empirically, MIOv3 achieves state-of-the-art performance on small-scale datasets (CIFAR-10/100, STL-10, Tiny ImageNet) and strong linear-evaluation results on ImageNet100/1K, outperforming several contrastive and non-contrastive methods, with ablations guiding the influence of temperature, training duration, batch size, and parameter count. The work also presents transfer-learning results on medical imaging and provides extensive eigenvalue analyses illustrating optimization dynamics and saddle-point behavior in SSL settings.

Abstract

Self-supervised contrastive learning frameworks have progressed rapidly over the last few years. In this paper, we propose a novel loss function for contrastive learning. We model our pre-training task as a binary classification problem to induce an implicit contrastive effect. We further improve the näive loss function after removing the effect of the positive-positive repulsion and incorporating the upper bound of the negative pair repulsion. Unlike existing methods, the proposed loss function optimizes the mutual information in positive and negative pairs. We also present a closed-form expression for the parameter gradient flow and compare the behaviour of self-supervised contrastive frameworks using Hessian eigenspectrum to analytically study their convergence. The proposed method outperforms SOTA self-supervised contrastive frameworks on benchmark datasets such as CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet. After 200 pretraining epochs with ResNet-18 as the backbone, the proposed model achieves an accuracy of 86.36%, 58.18%, 80.50%, and 30.87% on the CIFAR-10, CIFAR-100, STL-10, and Tiny-ImageNet datasets, respectively, and surpasses the SOTA contrastive baseline by 1.93%, 3.57%, 4.85%, and 0.33%, respectively. The proposed framework also achieves a state-of-the-art accuracy of 78.4% (200 epochs) and 65.22% (100 epochs) Top-1 Linear Evaluation accuracy on ImageNet100 and ImageNet1K datasets, respectively.
Paper Structure (42 sections, 1 theorem, 81 equations, 10 figures, 12 tables)

This paper contains 42 sections, 1 theorem, 81 equations, 10 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Literature Survey
  3. Methodology
  4. Motivation behind MIOv1 loss function
  5. Effect of Removing a Positive-Positive Repulsion
  6. Does optimizing an upper bound of negative pair repulsion result in better representation learning?
  7. Relation of Proposed loss and Mutual Information
  8. Do contrastive SSL methods converge?
  9. Experimental Details
  10. Datasets
  11. Implementation Details
  12. Results and Analysis
  13. Results on Small-Scale Datasets
  14. Results on Large-Scale Datasets
  15. Comparison with Contrastive Algorithms
  16. ...and 27 more sections

Key Result

Lemma 4.1

Let $(b_t)_{t\geq1},(\eta_t)_{t\geq1}$ be two non-negative sequences and $(a_t)_{t\geq 1}$ a sequence of vectors in a vector space $X$. Let $p \geq 1$ and assume $\sum_{t=1}^{\infty} \eta_t b_t^p < \infty$ and $\sum_{t=1}^{\infty} \eta_t = \infty$. Assume also that there exists $L \geq 0$ such that

Figures (10)

  • Figure 1: (a) Uniformity vs. Temperature, (b) Alignment vs. Temperature plot (c) Inter-class Uniformity vs Temperature, and (d) Accuracy vs Temperature plot at temperatures $\tau \in \{ 0.1, 0.2, 0.5\}$ for MIOv1, MIOv2 and MIOv3 on the CIFAR10 dataset.
  • Figure 2: Graphical Model pgmbook showing the dependency between two samples in a positive pair and the independency between two samples forming a negative pair. Here, $z$ and $z'$ are two different samples in a dataset. $t_1, t_2, t_3, t_4$ are randomly chosen transformations from the distribution $T$. $z_1$ and $z_2$ are obtained by applying $t_1$ and $t_2$ on $z$. $z_3$ and $z_4$ are obtained by applying $t_3$ and $t_4$ on $z'$.
  • Figure 3: Plot of eigenvalues of parameters of ResNet18, obtained after 200 epochs of pre-training on CIFAR10 and CIFAR100 datasets with different SSL frameworks, namely, SimCLR, DCL and MIOv3.
  • Figure 4: This figure shows how the feature vectors are obtained from the samples ($x_1, x_2, \hdots, x_N$) in a batch.
  • Figure 5: This figure shows how the pairings are obtained. The red cells indicate self-pairs, green cells indicate positive pairs, i.e., pairings between feature vectors of two augmented versions of the same sample, and blue cells indicate negative pairings, i.e. pairings between feature vectors of different samples.
  • ...and 5 more figures

Theorems & Definitions (1)

  • Lemma 4.1