Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitization

TL;DR

The paper investigates privacy risks in LLM-powered chatbots when handling sensitive PHI/PII, focusing on input regurgitation and prompt-induced sanitization to enforce HIPAA and GDPR compliance. It introduces a methodology of adding privacy-oriented prompts, evaluated on medical notes and hiring data, showing substantial leakage reductions (e.g., PHI leakage down to 26.4% of baseline) while preserving usable non-sensitive information. The study reveals non-uniform leakage across demographic subgroups and highlights biases in anonymization outcomes, underscoring both the promise and limits of prompt-based privacy controls. Overall, prompt-induced sanitization emerges as a practical, though not foolproof, approach to improve privacy in AI systems, with clear avenues for extending analysis to more domains and models.

Abstract

LLM-powered chatbots are becoming widely adopted in applications such as healthcare, personal assistants, industry hiring decisions, etc. In many of these cases, chatbots are fed sensitive, personal information in their prompts, as samples for in-context learning, retrieved records from a database, or as part of the conversation. The information provided in the prompt could directly appear in the output, which might have privacy ramifications if there is sensitive information there. As such, in this paper, we aim to understand the input copying and regurgitation capabilities of these models during inference and how they can be directly instructed to limit this copying by complying with regulations such as HIPAA and GDPR, based on their internal knowledge of them. More specifically, we find that when ChatGPT is prompted to summarize cover letters of a 100 candidates, it would retain personally identifiable information (PII) verbatim in 57.4% of cases, and we find this retention to be non-uniform between different subgroups of people, based on attributes such as gender identity. We then probe ChatGPT's perception of privacy-related policies and privatization mechanisms by directly instructing it to provide compliant outputs and observe a significant omission of PII from output.

Figures (9)

• Figure 1: Our experimental setup and methodology, where we first quantify ChatGPT's capability to copy and retain personally identifiable information (left). Then, we instruct ChatGPT to sanitize its output using k-anonymity, and to abide by privacy policies (HIPAA).
• Figure 2: This image showcases the utility analysis of skills and hireability of role in the hiring dataset (left) & of symptoms and diagnosis for the medical dataset (right).
• Figure 3: This figure demonstrates the utility analysis of patient identification and the retrieval of patient ID for the medical dataset. We notice our prompts are able to redact private information while still letting organizations fully utilize relevant attributes.
• Figure 4: This figure presents the utility analysis of Industry and Role retention in the Hiring Dataset. This allows us to validate that the dataset may still be viable for analytics post-anonymization, such as examining the number of applicants per industrial role.
• Figure 5: The above image shows the impact of retention of different genders post-anonymization in the hiring dataset. The decreasing trend suggests that our prompts are effective and able to anonymize gender.