Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models

Zongyu Wu, Minhua Lin, Zhiwei Zhang, Fali Wang, Xianren Zhang, Xiang Zhang, Suhang Wang

TL;DR

This work tackles privacy risks in large vision-language models by studying membership inference attacks targeting whether a specific image was used in training. It introduces ICIMIA, which exploits corruption robustness in embeddings for white-box attacks and robustness of text outputs under corruption for black-box attacks. Empirical results on VL-MIA/Flickr datasets with LLaVA-1.5 models show high AUCs (≈0.88 in white-box) and competitive black-box performance, outperforming several baselines. The findings reveal practical privacy risks in LVLM training data and highlight the need for privacy-preserving data practices and robust benchmarking in multimodal models.

Abstract

Large vision-language models (LVLMs) have demonstrated outstanding performance in many downstream tasks. However, LVLMs are trained on large-scale datasets, which can pose privacy risks if training images contain sensitive information. Therefore, it is important to detect whether an image is used to train the LVLM. Recent studies have investigated membership inference attacks (MIAs) against LVLMs, including detecting image-text pairs and single-modality content. In this work, we focus on detecting whether a target image is used to train the target LVLM. We design simple yet effective Image Corruption-Inspired Membership Inference Attacks (ICIMIA) against LVLMs, which are inspired by LVLM's different sensitivity to image corruption for member and non-member images. We first perform an MIA method under the white-box setting, where we can obtain the embeddings of the image through the vision part of the target LVLM. The attacks are based on the embedding similarity between the image and its corrupted version. We further explore a more practical scenario where we have no knowledge about target LVLMs and we can only query the target LVLMs with an image and a textual instruction. We then conduct the attack by utilizing the output text embeddings' similarity. Experiments on existing datasets validate the effectiveness of our proposed methods under those two different settings.

Image Corruption-Inspired Membership Inference Attacks against Large Vision-Language Models

TL;DR

This work tackles privacy risks in large vision-language models by studying membership inference attacks targeting whether a specific image was used in training. It introduces ICIMIA, which exploits corruption robustness in embeddings for white-box attacks and robustness of text outputs under corruption for black-box attacks. Empirical results on VL-MIA/Flickr datasets with LLaVA-1.5 models show high AUCs (≈0.88 in white-box) and competitive black-box performance, outperforming several baselines. The findings reveal practical privacy risks in LVLM training data and highlight the need for privacy-preserving data practices and robust benchmarking in multimodal models.

Abstract

Large vision-language models (LVLMs) have demonstrated outstanding performance in many downstream tasks. However, LVLMs are trained on large-scale datasets, which can pose privacy risks if training images contain sensitive information. Therefore, it is important to detect whether an image is used to train the LVLM. Recent studies have investigated membership inference attacks (MIAs) against LVLMs, including detecting image-text pairs and single-modality content. In this work, we focus on detecting whether a target image is used to train the target LVLM. We design simple yet effective Image Corruption-Inspired Membership Inference Attacks (ICIMIA) against LVLMs, which are inspired by LVLM's different sensitivity to image corruption for member and non-member images. We first perform an MIA method under the white-box setting, where we can obtain the embeddings of the image through the vision part of the target LVLM. The attacks are based on the embedding similarity between the image and its corrupted version. We further explore a more practical scenario where we have no knowledge about target LVLMs and we can only query the target LVLMs with an image and a textual instruction. We then conduct the attack by utilizing the output text embeddings' similarity. Experiments on existing datasets validate the effectiveness of our proposed methods under those two different settings.

Paper Structure

This paper contains 22 sections, 3 equations, 7 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Vision-Language Models
  4. Membership Inference Attack
  5. Preliminaries
  6. Large Vision-Language Models
  7. Threat Model
  8. Intuition for Performing MIA
  9. Method
  10. White-Box MIA via Image Similarity
  11. Black-box MIA via Text Similarity
  12. Experiments
  13. Experimental Setup
  14. White-Box MIA Performance
  15. Black-box MIA Performance
  16. ...and 7 more sections

Figures (7)

  • Figure 1: An illustration of the architecture of large vision-language models liu2024llava1.5.
  • Figure 2: A histogram of similarity scores of corrupted images' embeddings and original images' embeddings for member and non-member data.
  • Figure 3: A histogram of similarity scores of textual output embeddings between corrupted images and the original images for member and non-member data.
  • Figure 4: An illustration of the attack pipeline under two different settings.
  • Figure 5: Text similarity-based membership inference attack on VL-MIA/Flickr.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 1: Image Only MIA