Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memorization in Self-Supervised Learning Improves Downstream Generalization

Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, Franziska Boenisch

TL;DR

The paper addresses memorization in self-supervised learning (SSL), where labels are unavailable and memorization can leak private training data. It proposes SSLMem, a universal, label-free definition based on how well augmented views align across encoders trained with and without a given data point, avoiding dependence on any particular SSL objective. Through extensive experiments across architectures, SSL methods, and datasets, the authors show substantial memorization, especially for atypical samples, and demonstrate that memorization correlates with improved downstream generalization across tasks such as classification and semantic segmentation. They also explore differential privacy effects, showing that reducing memorization can harm downstream performance, highlighting privacy–utility trade-offs and suggesting practical data-selection strategies rooted in memorization signals. Overall, the work positions memorization as a fundamental aspect of SSL feature learning with meaningful implications for privacy and generalization.

Abstract

Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks.

Memorization in Self-Supervised Learning Improves Downstream Generalization

TL;DR

The paper addresses memorization in self-supervised learning (SSL), where labels are unavailable and memorization can leak private training data. It proposes SSLMem, a universal, label-free definition based on how well augmented views align across encoders trained with and without a given data point, avoiding dependence on any particular SSL objective. Through extensive experiments across architectures, SSL methods, and datasets, the authors show substantial memorization, especially for atypical samples, and demonstrate that memorization correlates with improved downstream generalization across tasks such as classification and semantic segmentation. They also explore differential privacy effects, showing that reducing memorization can harm downstream performance, highlighting privacy–utility trade-offs and suggesting practical data-selection strategies rooted in memorization signals. Overall, the work positions memorization as a fundamental aspect of SSL feature learning with meaningful implications for privacy and generalization.

Abstract

Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks.
Paper Structure (46 sections, 2 theorems, 6 equations, 11 figures, 17 tables)

This paper contains 46 sections, 2 theorems, 6 equations, 11 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. SSL.
  4. Membership Inference Attacks.
  5. Memorization.
  6. Towards Formalizing Memorization
  7. Preliminaries and Problem Setup
  8. Alignment and Memorization in SSL
  9. Experimental Evaluation
  10. Memorization over Different Architectures, SSL methods and Datasets
  11. Insights on the Memorization Score
  12. Memorized Data Points
  13. Memorization in SSL is Required for Downstream Generalization
  14. Classification.
  15. Semantic Segmentation.
  16. ...and 31 more sections

Key Result

Lemma 4

Given an encoder $f$ satisfying $c$-strong alignment over point $x$ and a test datapoint $z_i \in S_{\text{test}}$ which satisfies $\beta_i$-closeness to point $x$, $d(f(x), f(z_i)) \leq L \beta_i + c$

Figures (11)

  • Figure 1: Examples of data with different levels of memorization. Higher memorization scores indicate stronger memorization. We observe that outliers and atypical examples experience higher memorization than more standard samples. Results are obtained on a ViT-tiny, trained with MAE.
  • Figure 2: Insights into our memorization score. We train an MAE with VIT-tiny on CIFAR10. (a) We plot the alignment loss, computed with the $\ell_2$ distance, of the candidates (with respect to their augmentation) on encoder $f$ and encoder $g$. The color coding indicates the memorization score with higher scores indicating higher memorization. The lowest alignment loss on $f$ does not yield the highest memorization score, and high memorization can occur at a wide range of alignment loss values for $f$. (b) Training loss, downstream accuracy, and memorization over the course of training highlight that memorization is not just an effect of increasing/decreasing accuracy: while loss and accuracy stagnate after a few hundred epochs, memorization increases. (c) We report the memorization scores for 5000 data points from each subset $S_C$, $S_I$, $S_E$, and $S_S$. The encoders exhibit memorization indicated by significantly higher (lower) scores for $S_C$ ($S_I$) compared to $S_S$ or $S_E$.
  • Figure 3: The influence of memorization on downstream generalization (CIFAR10). We train an MAE model based on the VIT-tiny architecture on CIFAR10 and remove [500, 1k, 2k, 4k, 8k, 16k] most memorized vs. random data points from the encoder's training data. We measure downstream accuracy through linear probing on CIFAR10, CIFAR100, and STL10. The removal of memorized data points harms accuracy over all downstream tasks more than the removal of random data points.
  • Figure 4: Limiting memorization harms downstream accuracy.
  • Figure 5: Influence of the memorization threshold. Using the MAE-base model, we depict what fraction of data points from the respective candidate dataset would be classified as memorized by our definition when choosing the memorization threshold according to the number depicted on the x-axis.
  • ...and 6 more figures

Theorems & Definitions (7)

  • Definition 1: $c$-Strong Alignment
  • Definition 2: $\sigma$-overlap
  • Definition 3: $\beta$-close
  • Lemma 4
  • proof
  • Lemma 5
  • proof