The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Framework

TL;DR

The paper demonstrates a privacy risk where vision-language models infer sensitive and abstract attributes from multi-image personal photo collections, introducing the PAPI dataset and a HolmesEye agentic framework that fuses VLM perception with LLM reasoning. HolmesEye analyzes intra-image and inter-image cues, using iterative inference (extraction, analysis, summarization, inquiry, decision) to outperform baselines by up to 10.8 percentage points and exceed human accuracy on abstract attributes by 15.0%, while dramatically speeding profiling. The work provides a GDPR-compliant multi-image benchmark and a forensic-inspired method to quantify and study privacy risks, revealing practical implications for privacy protection in image-driven AI.

Abstract

Our research reveals a new privacy risk associated with the vision-language model (VLM) agentic framework: the ability to infer sensitive attributes (e.g., age and health information) and even abstract ones (e.g., personality and social traits) from a set of personal images, which we term "image private attribute profiling." This threat is particularly severe given that modern apps can easily access users' photo albums, and inference from image sets enables models to exploit inter-image relations for more sophisticated profiling. However, two main challenges hinder our understanding of how well VLMs can profile an individual from a few personal photos: (1) the lack of benchmark datasets with multi-image annotations for private attributes, and (2) the limited ability of current multimodal large language models (MLLMs) to infer abstract attributes from large image collections. In this work, we construct PAPI, the largest dataset for studying private attribute profiling in personal images, comprising 2,510 images from 251 individuals with 3,012 annotated privacy attributes. We also propose HolmesEye, a hybrid agentic framework that combines VLMs and LLMs to enhance privacy inference. HolmesEye uses VLMs to extract both intra-image and inter-image information and LLMs to guide the inference process as well as consolidate the results through forensic analysis, overcoming existing limitations in long-context visual reasoning. Experiments reveal that HolmesEye achieves a 10.8% improvement in average accuracy over state-of-the-art baselines and surpasses human-level performance by 15.0% in predicting abstract attributes. This work highlights the urgency of addressing privacy risks in image-based profiling and offers both a new dataset and an advanced framework to guide future research in this area.

Figures (4)

• Figure 1: HolmesEye develops VLM perception and LLM reasoning to extract private attributes from image collections, analyzing both individual images and cross-image patterns to profile users across four attribute domains.
• Figure 2: The PAPI dataset consists of 2,510 images collected from 251 volunteers across 31 occupations, with each individual represented by 10 photos. It includes 12 evaluation indicators across four categories: personal information, health condition, social attributes, and psychological traits.
• Figure 3: The process of inferring private information using the HolmesEye framework. Step 1 extracts individual image information. Step 2 analyzes inter-image relationships. Step 3 summarizes findings and identifies uninferable attributes. Step 4 inquiry missing indicators from all images. Step 5 consolidates results into comprehensive attribute profiles.
• Figure 4: The impact of the number of PAPI single-person images on the results.