Can Language Models be Instructed to Protect Personal Information?

TL;DR

PrivQA introduces a multimodal benchmark to quantify how well language and vision models can follow access-control instructions to protect personal information, while balancing utility. It defines Protected Groups and Protected Information, builds a textual and visual QA dataset, and proposes a self-moderation workflow that iteratively refines responses to improve privacy protection. The study finds that state-of-the-art models show improvements with self-moderation but remain biased and vulnerable to red-teaming and visual-prompt attacks, revealing a persistent privacy/utility trade-off. The work provides a publicly released dataset to standardize evaluation and motivate the development of more robust privacy protections in LLMs and vision-language models.

Abstract

Large multimodal language models have proven transformative in numerous applications. However, these models have been shown to memorize and leak pre-training data, raising serious user privacy and information security concerns. While data leaks should be prevented, it is also crucial to examine the trade-off between the privacy protection and model utility of proposed approaches. In this paper, we introduce PrivQA -- a multimodal benchmark to assess this privacy/utility trade-off when a model is instructed to protect specific categories of personal information in a simulated scenario. We also propose a technique to iteratively self-moderate responses, which significantly improves privacy. However, through a series of red-teaming experiments, we find that adversaries can also easily circumvent these protections with simple jailbreaking methods through textual and/or image inputs. We believe PrivQA has the potential to support the development of new models with improved privacy protections, as well as the adversarial robustness of these protections. We release the entire PrivQA dataset at https://llm-access-control.github.io/.

Figures (17)

• Figure 1: The PrivQA benchmark (§ \ref{['sec:dataset']}) consists of textual and visual question-answering tasks designed to assess the ability of multi-modal language models to protect private information. The model developers pre-define the Protected Groups of people (e.g., citizens of Italy) or types of information (e.g., geolocation) to be protected from the model. Models (e.g., GPT4, Flamingo) utilizes our proposed Self-Moderation technique (§ \ref{['sec:experiments']}) to selectively respond, abstaining Protected Groups while addressing questions to Control Groups (e.g., non-Italian public figures).
• Figure 2: Privacy-utility trade-off performance (protection score & Response F$_1$) on textual tasks from PrivQA for protected populations (left) and protected information (right). Baseline response model indicates Response F$_1$ without access control as a reference.
• Figure 3: Privacy-utility trade-off on visual tasks, on average of Citizenship and Geolocation.
• Figure 4: Protection score change over multiple self-authorization steps. GPT-series models benefit from additional steps of self-authorization.
• Figure 5: Entity popularity (est. by Wiki monthly pageviews) vs. sensitivity of the protected group, on average of protected populations.