Mutual Information guided Visual Contrastive Learning

TL;DR

The paper tackles the challenge of positive sample selection in self-supervised contrastive learning by introducing InfoAug, a mutual-information-guided augmentation that discovers cross-entity positives via twin patches whose trajectories share high mutual information. It combines traditional view-based augmentation with MI-aware cross-patch learning in a two-branch architecture, enabling simultaneous emphasis on view invariance and information-driven relationships. Through experiments across seven state-of-the-art frameworks on CIFAR-10, CIFAR-100, and STL-10, InfoAug yields consistent improvements and demonstrates the utility of MI-based patch selection over random baselines. The work also discusses practical limitations in dynamic in-the-wild datasets and outlines future directions, including enhanced patch representations and temporal integration for a more unified contrastive learning paradigm.

Abstract

Representation learning methods utilizing the InfoNCE loss have demonstrated considerable capacity in reducing human annotation effort by training invariant neural feature extractors. Although different variants of the training objective adhere to the information maximization principle between the data and learned features, data selection and augmentation still rely on human hypotheses or engineering, which may be suboptimal. For instance, data augmentation in contrastive learning primarily focuses on color jittering, aiming to emulate real-world illumination changes. In this work, we investigate the potential of selecting training data based on their mutual information computed from real-world distributions, which, in principle, should endow the learned features with better generalization when applied in open environments. Specifically, we consider patches attached to scenes that exhibit high mutual information under natural perturbations, such as color changes and motion, as positive samples for learning with contrastive loss. We evaluate the proposed mutual-information-informed data augmentation method on several benchmarks across multiple state-of-the-art representation learning frameworks, demonstrating its effectiveness and establishing it as a promising direction for future research.

Figures (5)

• Figure 1: This figure shows the general pipeline of InfoAug. Through video tracking, we estimate the pairwise mutual information between patches attached to a same scene, which we base on to construct a "twin patch" dictionary. While training, besides traditional data augmentation($T1, T2$), we use twin patch dictionary to form another positive pair, thereby injecting mutual information awareness into the model, towards a more unifed data augmentation paradigm.
• Figure 2: This figure shows a two-branch learning formulation adopted by InfoAug, where the backbone use weight-sharing while the two projection heads is independently updated, which aims for a decoupling of the two learning objectives: 1.view-invariant and 2.mutual information aware.
• Figure 3: Ablation study on the weighted factor
• Figure 4: This is the figure from micomplex which gives a clear definition of how "3KL" estimate density based on the k-th distance with $L_\infty-{norm}$
• Figure 5: The mutual information empirical estimation time with "3KL" methodology on different numbers of observation