Membership Inference Attacks against Large Vision-Language Models

TL;DR

This study introduces the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection and proposes a novel MIA pipeline specifically designed for token-level image detection.

Abstract

Large vision-language models (VLLMs) exhibit promising capabilities for processing multi-modal tasks across various application scenarios. However, their emergence also raises significant data security concerns, given the potential inclusion of sensitive information, such as private photos and medical records, in their training datasets. Detecting inappropriately used data in VLLMs remains a critical and unresolved issue, mainly due to the lack of standardized datasets and suitable methodologies. In this study, we introduce the first membership inference attack (MIA) benchmark tailored for various VLLMs to facilitate training data detection. Then, we propose a novel MIA pipeline specifically designed for token-level image detection. Lastly, we present a new metric called MaxRényi-K%, which is based on the confidence of the model output and applies to both text and image data. We believe that our work can deepen the understanding and methodology of MIAs in the context of VLLMs. Our code and datasets are available at https://github.com/LIONS-EPFL/VL-MIA.

Figures (5)

• Figure 1: MIAs against VLLMs. Top: Our image detection pipeline: In the generation stage, we feed the image and instruction to the target model to obtain a description; then during the inference stage, we input the image, instruction, and generated description to the model, and extract the logits slices to calculate metrics. Bottom:MaxRényi-K% metric: we first get the Rényi entropy of each token position, then select the largest $k\%$ tokens and calculate the average Rényi entropy.
• Figure 2: Ablation study (a) on max_new_tokens with MaxRényi-10%. Allowing VLLMs to generate longer descriptions can increase the AUC of "desp" slices, but we encounter a plateau when max_new_tokens equals 128. (b) on image MIAs against corrupted versions of VL-MIA/Flickr with MaxRényi-K% ($\alpha$ = 0.5). Three levels of corruption are applied to the images: Marginal, Moderate, and Severe. The dotted line indicates the AUC on raw images without corruption.
• Figure 3: Different slices in the prompt.
• Figure 4: A schematic of VLLM.
• Figure 5: Examples of new geometry datasets (top), new password datasets (bottom)..