Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

Jie Zhu, Jirong Zha, Ding Li, Leye Wang

TL;DR

This work addresses privacy risks in visual self-supervised learning by introducing PartCrop, a unified membership inference method that does not require knowledge of the training recipe and leverages part-aware representations. PartCrop operates in a black-box, multi-paradigm SSL setting by extracting image and part-feature responses, forming membership features via KL-divergence-based distributions, and training a lightweight attacker. Extensive evaluations across MAE, DINO, and MoCo on CIFAR-10, CIFAR-100, and TinyImageNet show PartCrop outperforms baselines and generalizes to shadow settings, with ablations clarifying the roles of feature type, crop count, and crop scale. The paper also analyzes defenses (early stop, DP, shrinking crop scale range) and conducts large-scale scaling studies, culminating in PartCrop-v2, which improves attacker stability through activation and normalization refinements and demonstrates broader applicability across diverse SSL paradigms and datasets.

Abstract

Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses within the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Finally, besides prototype testing on toy visual encoders and small-scale image datasets, we quantitatively study the impacts of scaling from both data and model aspects in a realistic scenario and propose a scalable PartCrop-v2 by introducing two structural improvements to PartCrop. Our code is at https://github.com/JiePKU/PartCrop.

A Unified and Scalable Membership Inference Method for Visual Self-supervised Encoder via Part-aware Capability

TL;DR

This work addresses privacy risks in visual self-supervised learning by introducing PartCrop, a unified membership inference method that does not require knowledge of the training recipe and leverages part-aware representations. PartCrop operates in a black-box, multi-paradigm SSL setting by extracting image and part-feature responses, forming membership features via KL-divergence-based distributions, and training a lightweight attacker. Extensive evaluations across MAE, DINO, and MoCo on CIFAR-10, CIFAR-100, and TinyImageNet show PartCrop outperforms baselines and generalizes to shadow settings, with ablations clarifying the roles of feature type, crop count, and crop scale. The paper also analyzes defenses (early stop, DP, shrinking crop scale range) and conducts large-scale scaling studies, culminating in PartCrop-v2, which improves attacker stability through activation and normalization refinements and demonstrates broader applicability across diverse SSL paradigms and datasets.

Abstract

Self-supervised learning shows promise in harnessing extensive unlabeled data, but it also confronts significant privacy concerns, especially in vision. In this paper, we perform membership inference on visual self-supervised models in a more realistic setting: self-supervised training method and details are unknown for an adversary when attacking as he usually faces a black-box system in practice. In this setting, considering that self-supervised model could be trained by completely different self-supervised paradigms, e.g., masked image modeling and contrastive learning, with complex training details, we propose a unified membership inference method called PartCrop. It is motivated by the shared part-aware capability among models and stronger part response on the training data. Specifically, PartCrop crops parts of objects in an image to query responses within the image in representation space. We conduct extensive attacks on self-supervised models with different training protocols and structures using three widely used image datasets. The results verify the effectiveness and generalization of PartCrop. Moreover, to defend against PartCrop, we evaluate two common approaches, i.e., early stop and differential privacy, and propose a tailored method called shrinking crop scale range. The defense experiments indicate that all of them are effective. Finally, besides prototype testing on toy visual encoders and small-scale image datasets, we quantitatively study the impacts of scaling from both data and model aspects in a realistic scenario and propose a scalable PartCrop-v2 by introducing two structural improvements to PartCrop. Our code is at https://github.com/JiePKU/PartCrop.
Paper Structure (37 sections, 9 equations, 12 figures, 13 tables)

This paper contains 37 sections, 9 equations, 12 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Threat Model
  4. Method
  5. Experimental Setting
  6. Self-supervised Model
  7. Dataset and Metric
  8. Baseline
  9. Implementation Details
  10. Experiment
  11. Results in the Partial Setting
  12. Ablation Study
  13. Merbership Feature
  14. Crop Number
  15. Crop Scale
  16. ...and 22 more sections

Figures (12)

  • Figure 1: In a part retrieval experiment, the cropped part in red boundingbox (i.e., a human head) is used to query in a large image dataset. DeiT touvron2021training uses supervised learning. MAE he2022masked and CAE chen2022context are masked image modeling based methods. DINO caron2021emerging and MoCo v3 chen2021empirical are contrastive learning based methods. iBOT zhou2021ibot combines the two paradigms. This figure is borrowed from zhu2023understanding. We refer readers of interest to zhu2023understanding.
  • Figure 2: Part response visualization on MAE (masked), DINO (contrastive), and MoCo (contrastive). Images are from Tinyimagenet tinyimagenet_le2015tiny. For each image containing a chair or dog, we manually crop it to obtain the part (chair seat or dog muzzle) in the red box and resize it to suitable size. Next, we calculate the cosine similarity between each vector of the feature map of entire image and the part feature vector from the encoder, sorting the scores in descending order. (a), (b), (c) are the results of chair image and chair seat part on MAE, DINO, and MoCo, respectively. (d), (e), (f) are the results of dog image and dog muzzle part on MAE, DINO, and MoCo, respectively.
  • Figure 3: An overview of PartCrop.
  • Figure 4: Ablation study on crop number. We consider four different crop number i.e., 32, 64, 128, and 256.
  • Figure 5: Ablation study on crop scale. We consider five different crop scales i.e., $(0.08,\; 0.1)$, $(0.08,\; 0.2)$, $(0.08,\; 0.3)$, $(0.01,\; 0.03)$, and $(0.5,\; 1.0)$.
  • ...and 7 more figures