Where Did Your Model Learn That? Label-free Influence for Self-supervised Learning

TL;DR

Influence-SSL introduces a label-free data attribution framework for self-supervised learning by measuring how training points affect their own representations through augmentation-induced stability, grounded in an invariance-distinctiveness trade-off. The authors derive a theoretical foundation in a linear setting, showing that influence corresponds to augmentation sensitivity and representation changes, and validate the approach across SimCLR, BYOL, and Barlow Twins on CIFAR-10/100 and FairFace, demonstrating stability and practical utility. Key findings include that high-influence points can hinder SSL learning via background biases, and that removing such points can improve downstream performance, while influence analysis also enables detection of semantic duplicates and fairness-related biases. The work provides a data-centric tool for SSL analysis with implications for data curation, robustness, and fairness in large-scale representation learning.

Abstract

Self-supervised learning (SSL) has revolutionized learning from large-scale unlabeled datasets, yet the intrinsic relationship between pretraining data and the learned representations remains poorly understood. Traditional supervised learning benefits from gradient-based data attribution tools like influence functions that measure the contribution of an individual data point to model predictions. However, existing definitions of influence rely on labels, making them unsuitable for SSL settings. We address this gap by introducing Influence-SSL, a novel and label-free approach for defining influence functions tailored to SSL. Our method harnesses the stability of learned representations against data augmentations to identify training examples that help explain model predictions. We provide both theoretical foundations and empirical evidence to show the utility of Influence-SSL in analyzing pre-trained SSL models. Our analysis reveals notable differences in how SSL models respond to influential data compared to supervised models. Finally, we validate the effectiveness of Influence-SSL through applications in duplicate detection, outlier identification and fairness analysis. Code is available at: \url{https://github.com/cryptonymous9/Influence-SSL}.

Figures (16)

• Figure 1: Illustration of scenarios for varying influence levels for self-supervised learning: (1) Hard negative examples, which challenge the model's decision boundary, should have high influence. (2) Atypical examples, representing rare or outlier data, should also exhibit high influence. (3) Near-duplicate examples should maintain low influence due to their redundant nature and low contribution to learning.
• Figure 2: Correlation of Influence-SSL scores for two independent runs on SimCLR, Barlow Twins, and BYOL. Consistently high correlations are observed between influence scores computed using different initializations.
• Figure 3: t-SNE projection of CIFAR-10 training images shows all examples (left), the 2,000 lowest (middle), and highest (right) influence scores. Low-influence images cluster tightly, while high-influence ones are dispersed.
• Figure 4: Five lowest influential images of the CIFAR-10 'automobile' class identified using BYOL, SimCLR, and Barlow Twins. Duplicate images are highlighted in red color.
• Figure 5: Visualization of the 12 highest and 12 lowest influential examples in CIFAR-100, showing that the highest influence images predominantly feature uniform backgrounds.

Theorems & Definitions (37)

• Theorem 1: Influence-SSL Characterization
• Proposition 1: Structural Properties of Influence-SSL
• Definition 1: Expected SSL Influence
• Proposition 2: Influence Deviation Characterization
• Definition 2: Two-Layer Linear Network
• Lemma 1: Gradients
• proof
• Lemma 2: Hessian
• proof
• Definition 3: SSL Setting