Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild

Niloofar Mireshghallah, Maria Antoniak, Yash More, Yejin Choi, Golnoosh Farnadi

TL;DR

This work investigates personal disclosures in real-world human-LLM conversations, focusing on PII leakage and sensitive topics in the WildChat dataset. It introduces a task-based and a sensitive-topic taxonomy, coupled with automatic annotations, to quantify what users disclose and under which contexts. The study finds that PII appears in a majority of queries and that many sensitive disclosures lie outside traditional PII categories, raising privacy concerns and highlighting limitations of current detectors. The authors advocate privacy-preserving designs, contextual nudges, and local models to mitigate disclosure risks in practical deployments.

Abstract

Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users' AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.

Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild

TL;DR

This work investigates personal disclosures in real-world human-LLM conversations, focusing on PII leakage and sensitive topics in the WildChat dataset. It introduces a task-based and a sensitive-topic taxonomy, coupled with automatic annotations, to quantify what users disclose and under which contexts. The study finds that PII appears in a majority of queries and that many sensitive disclosures lie outside traditional PII categories, raising privacy concerns and highlighting limitations of current detectors. The authors advocate privacy-preserving designs, contextual nudges, and local models to mitigate disclosure risks in practical deployments.

Abstract

Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users' AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.
Paper Structure (31 sections, 8 figures, 4 tables)

This paper contains 31 sections, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Data and Methods
  3. Data
  4. Task Annotation
  5. Task Distribution
  6. How much detectable PII do users share?
  7. PII Detection
  8. Detected PII Distribution
  9. Is PII detection sufficient for privacy?
  10. Sensitive Topic Detection
  11. Discovering sensitive topics
  12. Where does PII detection fall short?
  13. In what conversational contexts are sensitive topics mentioned?
  14. Discussion
  15. Design implications
  16. ...and 16 more sections

Figures (8)

  • Figure 1: Real examples of personal disclosures that we found within user-chatbot conversations in the WildChat dataset. We have altered names and other PII to preserve privacy. We can see that users disclose identifiable information about themselves and others to ChatGPT, and in the process, to the publicly available WildChat dataset. We were able to de-identify each of these examples.
  • Figure 2: We plot the distribution of tasks over (a) a random sample of 5k WildChat conversations, filtered to one conversation per IP address, (b) a random sample of 1k WildChat conversations IP address or prefix filtering, and (c) a random sample of 1k ShareGPT conversations.
  • Figure 3: Fine-grained PII entities across WildChat and ShareGPT, using the Azure AI Language service for annotation. We keep the IBAN (international banking) category despite a high error rate because the detected strings are still PII (mostly API tokens).
  • Figure 4: Relationship between task annotations of WildChat queries and detected PII.
  • Figure 5: Relationship between sensitive topic annotations of WildChat queries and different kinds of detected PII.
  • ...and 3 more figures