PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility

TL;DR

PII-VisBench introduces a visibility-aware benchmark for evaluating PII safety in vision-language systems, leveraging a continuum of subject visibility across 200 individuals and 20 PII attributes to yield 4000 probes evaluated on 18 open-source VLMs. The study uses Refusal Rate and Conditional PII Disclosure Rate, supplemented by paraphrase and jailbreak stress tests, to reveal a consistent high-visibility privacy gap and substantial model heterogeneity in privacy alignment. Across architectures, structured PII tends to be refused more reliably than demographic traits, and safety performance interacts with model generation and size in non-monotonic ways. The work highlights the need for visibility-aware safety training and evaluation strategies to ensure privacy protections across the full spectrum of online presence in real-world deployments.

Abstract

Vision Language Models (VLMs) are increasingly integrated into privacy-critical domains, yet existing evaluations of personally identifiable information (PII) leakage largely treat privacy as a static extraction task and ignore how a subject's online presence--the volume of their data available online--influences privacy alignment. We introduce PII-VisBench, a novel benchmark containing 4000 unique probes designed to evaluate VLM safety through the continuum of online presence. The benchmark stratifies 200 subjects into four visibility categories: high, medium, low, and zero--based on the extent and nature of their information available online. We evaluate 18 open-source VLMs (0.3B-32B) based on two key metrics: percentage of PII probing queries refused (Refusal Rate) and the fraction of non-refusal responses flagged for containing PII (Conditional PII Disclosure Rate). Across models, we observe a consistent pattern: refusals increase and PII disclosures decrease (9.10% high to 5.34% low) as subject visibility drops. We identify that models are more likely to disclose PII for high-visibility subjects, alongside substantial model-family heterogeneity and PII-type disparities. Finally, paraphrasing and jailbreak-style prompts expose attack and model-dependent failures, motivating visibility-aware safety evaluation and training interventions.

Figures (8)

• Figure 1: Motivating example of privacy responses in VLMs. When prompted with a Zero-visibility subject (AI-generated image), closed-source models (GPT-5.1, Gemini 3 Pro) refuse while the open-source Phi3.5 4B identifies the lack of information but LLaVA1.5 13B produces a specific address.
• Figure 2: Impact of model generation on refusal behavior.
• Figure 3: Impact of model paramaters on refusal behavior.
• Figure 4: Average refusal rates (RR %) across VLMs for high-visibility subjects under original and paraphrased prompts, with smoothed trend lines highlighting overall shifts pattern. Here, Phi = Phi3.5, L = Llama3.2, G = Gemma3, IVL = InternVL, LV = LLaVA, Q = Qwen, sVLM = SmolVLM.
• Figure 5: Violin and box plot comparison of search-result visibility (log$_{10}$ scale) for the high-visibility and medium-visibility groups. The dashed horizontal line indicates the divider used to separate the two curated groups: $6{,}870{,}000$ results, equal to the maximum count observed in the medium-visibility set. Points show individual observations with jitter for readability.