Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage

Rui Xin, Niloofar Mireshghallah, Shuyue Stella Li, Michael Duan, Hyunwoo Kim, Yejin Choi, Yulia Tsvetkov, Sewoong Oh, Pang Wei Koh

TL;DR

This work confronts the inadequacy of surface-level PII removal for textual data by introducing a semantic privacy framework that evaluates re-identification risk when sanitized text is released with auxiliary information. It decomposes records into atomic claims, links auxiliary data to sanitized documents with a dense GRIT retriever, and measures leakage via a semantic distance μ, while tracking data utility. Key findings show that commercial PII tools leave substantial information leakage (e.g., 74%), synthesized data under non-DP conditions also leak, and DP synthesis provides stronger privacy but at a notable cost to utility and text quality. The results underscore a false sense of privacy in many current methods and call for robust, semantics-aware sanitization techniques that maintain downstream usefulness for real-world tasks.

Abstract

Sanitizing sensitive text data typically involves removing personally identifiable information (PII) or generating synthetic data under the assumption that these methods adequately protect privacy; however, their effectiveness is often only assessed by measuring the leakage of explicit identifiers but ignoring nuanced textual markers that can lead to re-identification. We challenge the above illusion of privacy by proposing a new framework that evaluates re-identification attacks to quantify individual privacy risks upon data release. Our approach shows that seemingly innocuous auxiliary information -- such as routine social activities -- can be used to infer sensitive attributes like age or substance use history from sanitized data. For instance, we demonstrate that Azure's commercial PII removal tool fails to protect 74\% of information in the MedQA dataset. Although differential privacy mitigates these risks to some extent, it significantly reduces the utility of the sanitized text for downstream tasks. Our findings indicate that current sanitization techniques offer a \textit{false sense of privacy}, highlighting the need for more robust methods that protect against semantic-level information leakage.

A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage

TL;DR

This work confronts the inadequacy of surface-level PII removal for textual data by introducing a semantic privacy framework that evaluates re-identification risk when sanitized text is released with auxiliary information. It decomposes records into atomic claims, links auxiliary data to sanitized documents with a dense GRIT retriever, and measures leakage via a semantic distance μ, while tracking data utility. Key findings show that commercial PII tools leave substantial information leakage (e.g., 74%), synthesized data under non-DP conditions also leak, and DP synthesis provides stronger privacy but at a notable cost to utility and text quality. The results underscore a false sense of privacy in many current methods and call for robust, semantics-aware sanitization techniques that maintain downstream usefulness for real-world tasks.

Abstract

Sanitizing sensitive text data typically involves removing personally identifiable information (PII) or generating synthetic data under the assumption that these methods adequately protect privacy; however, their effectiveness is often only assessed by measuring the leakage of explicit identifiers but ignoring nuanced textual markers that can lead to re-identification. We challenge the above illusion of privacy by proposing a new framework that evaluates re-identification attacks to quantify individual privacy risks upon data release. Our approach shows that seemingly innocuous auxiliary information -- such as routine social activities -- can be used to infer sensitive attributes like age or substance use history from sanitized data. For instance, we demonstrate that Azure's commercial PII removal tool fails to protect 74\% of information in the MedQA dataset. Although differential privacy mitigates these risks to some extent, it significantly reduces the utility of the sanitized text for downstream tasks. Our findings indicate that current sanitization techniques offer a \textit{false sense of privacy}, highlighting the need for more robust methods that protect against semantic-level information leakage.
Paper Structure (69 sections, 1 equation, 5 figures, 11 tables)

This paper contains 69 sections, 1 equation, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Privacy Metric
  4. Problem statement.
  5. Linking method $L$.
  6. Semantic distance metric $\mu$.
  7. Lexical baseline.
  8. Experiment Setup
  9. Datasets.
  10. Utility metrics.
  11. Data sanitization techniques.
  12. Experiment Results
  13. The Privacy-Utility Tradeoff: Identifier Removal and Data Synthesis
  14. The Privacy-Utility Tradeoff: Data Synthesis with Differential Privacy
  15. Analysis
  16. ...and 54 more sections

Figures (5)

  • Figure 1: Our privacy evaluation framework overview. For a given sanitization method, we obtain the (2) sanitized dataset by applying the method to the (1) original dataset. In the linking stage, we use (3) auxiliary information to find potential matches in the sanitized dataset using a semantic retriever to obtain the (5) linked record. Next, in the semantic scoring stage, we analyze the linked record against the corresponding (4) original record to identify semantic information leakage. In the linked record, text highlighted in yellow indicates detected leakage, and in cyan indicates content used for the linking process. The framework calculates utility scores to measure the practical value of the sanitized dataset and a privacy score based on the detected information leakage.
  • Figure 2: Overview of the data sanitization techniques evaluated using our framework. We evaluate two main categories: identifier removal methods and data synthesis methods. Identifier removal methods operate at the sample level, maintaining a one-to-one correspondence between original and sanitized records. In contrast, data synthesis methods operate at the dataset level, where each sanitized record may derive information from multiple original records.
  • Figure 3: Privacy scores to the number of claims available to the attacker across different sanitization methods (§\ref{['sec:san_techniques']}). Sanitization methods are introduced in §\ref{['sec:san_techniques']}. In particular, Span Sanitization refers to sanitization method proposed by dou2023reducing, and Iterative Anonymization refers to the technique proposed by staab2024large. The Use Aux Information row quantifies the information overlap between auxiliary information provided to attackers and the remaining document content.
  • Figure 4: Distribution of leaked sensitive categories for each of the sanitization methods (§\ref{['sec:san_techniques']}) on the MedQA and WildChat dataset. Span Sanitization refers to sanitization method proposed by dou2023reducing, and Iterative Anonymization refers to the technique proposed by staab2024large.
  • Figure 5: Distribution of privacy scores for different sanitization methods (§\ref{['sec:san_techniques']}) used in the study. Span Sanitization refers to sanitization method proposed by dou2023reducing, and Iterative Anonymization refers to the technique proposed by staab2024large.