Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models

Ryoto Miyamoto, Xin Fan, Fuyuko Kido, Tsuneo Matsumoto, Hayato Yamana

TL;DR

OpenLVLM-MIA exposes the critical problem that many MIA evaluations for LVLMs conflate dataset biases with true membership signals. By constructing a bias-controlled, ground-truth-annotated 6,000-image benchmark and evaluating a transparent OpenCLIP-LLaVA pipeline across three training stages, the study shows that state-of-the-art MIAs perform at chance under proper controls. The work highlights the need for distribution auditing, multimodal-specific attack strategies, and fully disclosed training data to enable reliable privacy assessments. It provides public datasets, tooling, and models to foster reproducibility and guide future privacy-preserving developments in vision-language systems.

Abstract

OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods approached chance-level. OpenLVLM-MIA, designed to be transparent and unbiased benchmark, clarifies certain limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.

OpenLVLM-MIA: A Controlled Benchmark Revealing the Limits of Membership Inference Attacks on Large Vision-Language Models

TL;DR

OpenLVLM-MIA exposes the critical problem that many MIA evaluations for LVLMs conflate dataset biases with true membership signals. By constructing a bias-controlled, ground-truth-annotated 6,000-image benchmark and evaluating a transparent OpenCLIP-LLaVA pipeline across three training stages, the study shows that state-of-the-art MIAs perform at chance under proper controls. The work highlights the need for distribution auditing, multimodal-specific attack strategies, and fully disclosed training data to enable reliable privacy assessments. It provides public datasets, tooling, and models to foster reproducibility and guide future privacy-preserving developments in vision-language systems.

Abstract

OpenLVLM-MIA is a new benchmark that highlights fundamental challenges in evaluating membership inference attacks (MIA) against large vision-language models (LVLMs). While prior work has reported high attack success rates, our analysis suggests that these results often arise from detecting distributional bias introduced during dataset construction rather than from identifying true membership status. To address this issue, we introduce a controlled benchmark of 6{,}000 images where the distributions of member and non-member samples are carefully balanced, and ground-truth membership labels are provided across three distinct training stages. Experiments using OpenLVLM-MIA demonstrated that the performance of state-of-the-art MIA methods approached chance-level. OpenLVLM-MIA, designed to be transparent and unbiased benchmark, clarifies certain limitations of MIA research on LVLMs and provides a solid foundation for developing stronger privacy-preserving techniques.
Paper Structure (51 sections, 2 equations, 1 figure, 5 tables)

This paper contains 51 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Escalating Privacy Risks
  3. Limitations of Existing MIA Research
  4. Motivation and Goal
  5. Contributions
  6. Related Work
  7. Membership Inference Attacks
  8. MIA Benchmark Datasets for Vision--Language Models
  9. VL-MIA Dataset
  10. Other Datasets
  11. Effects of Training Stages on MIAs
  12. Preliminary
  13. Problem Formulation
  14. Distribution Bias Problem
  15. Proposed Benchmark Model: OpenCLIP-LLaVA
  16. ...and 36 more sections

Figures (1)

  • Figure 1: LDA+PCA 3D visualization of the embedding space. dim1 is the supervised LDA discriminant axis; dim2--3 are the top two PCA components in the subspace orthogonal to the LDA axis. VL-MIA shows noticeable separation, while OpenLVLM-MIA exhibits considerable overlap.