The Pervasive Blind Spot: Benchmarking VLM Inference Risks on Everyday Personal Videos
Shuning Zhang, Zhaoxin Li, Changxi Wen, Ying Ma, Simin Li, Gengrui Zhang, Ziyi Zhang, Yibo Meng, Hantao Zhao, Xin Yi, Hewu Li
TL;DR
This work benchmarks inferential privacy risks posed by Vision-Language Models on everyday personal videos using a crowdsourced dataset of 508 videos from 58 participants, comparing VLMs against human baselines. It demonstrates a paradigm shift from static object recognition to temporal, behavioral inference, with VLMs achieving superhuman accuracy under certain prompts and frame strategies. However, model explanations are often unreliable, and confounding ambient objects can mislead attribution, underscoring the need for grounded explainability and platform-level risk auditing. The findings highlight the urgency of rethinking privacy governance for pervasive video data and evaluating countermeasures that can robustly protect users in real-world deployment.
Abstract
The proliferation of Vision-Language Models (VLMs) introduces profound privacy risks from personal videos. This paper addresses the critical yet unexplored inferential privacy threat, the risk of inferring sensitive personal attributes over the data. To address this gap, we crowdsourced a dataset of 508 everyday personal videos from 58 individuals. We then conducted a benchmark study evaluating VLM inference capabilities against human performance. Our findings reveal three critical insights: (1) VLMs possess superhuman inferential capabilities, significantly outperforming human evaluators, leveraging a shift from object recognition to behavioral inference from temporal streams. (2) Inferential risk is strongly correlated with factors such as video characteristics and prompting strategies. (3) VLM-driven explanation towards the inference is unreliable, as we revealed a disconnect between the model-generated explanations and evidential impact, identifying ubiquitous objects as misleading confounders.