Little Data, Big Impact: Privacy-Aware Visual Language Models via Minimal Tuning

TL;DR

This work investigates privacy-awareness in Visual Language Models (VLMs) and identifies major gaps in recognizing privacy-sensitive content. It introduces two GDPR-aligned privacy benchmarks, PrivBench and PrivBench-H, and a privacy-tuning dataset, PrivTune, to enable targeted instruction tuning. The authors demonstrate that privacy-tuning on as few as 100 samples yields substantial improvements across privacy benchmarks—surpassing strong baselines like GPT-4 in several settings—while incurring only modest impact on non-privacy tasks. They also expose label noise in existing privacy datasets and show strong generalization to unseen privacy categories and multilingual contexts. Finally, they illustrate practical applicability by analyzing privacy across large image collections (e.g., Places365), highlighting the potential for scalable privacy screening in real-world deployments.

Abstract

As Visual Language Models (VLMs) become increasingly embedded in everyday applications, ensuring they can recognize and appropriately handle privacy-sensitive content is essential. We conduct a comprehensive evaluation of ten state-of-the-art VLMs and identify limitations in their understanding of visual privacy. Existing datasets suffer from label inconsistencies, limiting their reliability. To address this, we introduce two compact, high-quality benchmarks, PrivBench and PrivBench-H, that focus on commonly recognized privacy categories aligned with the General Data Protection Regulation (GDPR). Additionally, we present PrivTune, an instruction-tuning dataset specifically curated to improve privacy sensitivity. We obtain a Privacy VLM by fine-tuning an off-the-shelf VLM on only 100 samples from PrivTune, which leads to substantial gains on all benchmarks, surpassing GPT-4, while maintaining strong performance on other tasks. Our findings show that privacy-awareness in VLMs can be substantially improved with minimal data and careful dataset design, setting the stage for safer, more privacy-aligned AI systems.

Figures (10)

• Figure 1: Privacy-tuning Overview and Benchmark Results. From left to right: (i) our privacy-tuning pipeline, (ii) a qualitative example from the tuned model, (iii) Matthews Correlation Coefficient (MCC↑) comparison of our Privacy VLM with state-of-the-art VLMs on PrivBench and VISPR.
• Figure 2: Examples from the PrivTune dataset: This figure shows sample privacy-aware dialogues, each paired with human ground-truth labels and GPT-4-generated conversations. Images are blurred for visualisation.
• Figure 3: Common Labelling Errors and Limited Diversity in Biv-Priv and PrivAlert Datasets. Left: PrivAlert mislabels images containing people (blurring: ours) as non-private, while labeling dolls and paintings as private. Center: Repeated objects (17 of 56 images) within the 'pill bottle' class of Biv-Priv, illustrating limited diversity. Right: Biv-Priv labeling errors including black screens, empty sheets, and object-free images incorrectly labeled as private. Labels assigned by datasets appear at the bottom-right of each image.
• Figure 4: Human Evaluation on Privacy Datasets. The figure displays the accuracy and inter-rater agreement (Fleiss’ Kappa) for PrivAlert, Biv-Priv, VISPR and PrivBench.
• Figure 5: Samples from the VISPR dataset. Examples of privacy attributes (e.g., hair color) that are insufficient on their own to identify individuals; class labels are shown in the top-left corner.