Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CLIOPATRA: Extracting Private Information from LLM Insights

Meenatchi Sundaram Muthu Selva Annamalai, Emiliano De Cristofaro, Peter Kairouz

TL;DR

It is shown that current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems, and that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks.

Abstract

As AI assistants become widely used, privacy-aware platforms like Anthropic's Clio have been introduced to generate insights from real-world AI use. Clio's privacy protections rely on layering multiple heuristic techniques together, including PII redaction, clustering, filtering, and LLM-based privacy auditing. In this paper, we put these claims to the test by presenting CLIOPATRA, the first privacy attack against "privacy-preserving" LLM insight systems. The attack involves a realistic adversary that carefully designs and inserts malicious chats into the system to break multiple layers of privacy protections and induce the leakage of sensitive information from a target user's chat. We evaluated CLIOPATRA on synthetically generated medical target chats, demonstrating that an adversary who knows only the basic demographics of a target user and a single symptom can successfully extract the user's medical history in 39% of cases by just inspecting Clio's output. Furthermore, CLIOPATRA can reach close to 100% when Clio is configured with other state-of-the-art models and the adversary's knowledge of the target user is increased. We also show that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks. Our findings indicate that even when layered, current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems.

CLIOPATRA: Extracting Private Information from LLM Insights

TL;DR

It is shown that current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems, and that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks.

Abstract

As AI assistants become widely used, privacy-aware platforms like Anthropic's Clio have been introduced to generate insights from real-world AI use. Clio's privacy protections rely on layering multiple heuristic techniques together, including PII redaction, clustering, filtering, and LLM-based privacy auditing. In this paper, we put these claims to the test by presenting CLIOPATRA, the first privacy attack against "privacy-preserving" LLM insight systems. The attack involves a realistic adversary that carefully designs and inserts malicious chats into the system to break multiple layers of privacy protections and induce the leakage of sensitive information from a target user's chat. We evaluated CLIOPATRA on synthetically generated medical target chats, demonstrating that an adversary who knows only the basic demographics of a target user and a single symptom can successfully extract the user's medical history in 39% of cases by just inspecting Clio's output. Furthermore, CLIOPATRA can reach close to 100% when Clio is configured with other state-of-the-art models and the adversary's knowledge of the target user is increased. We also show that existing ad hoc mitigations, such as LLM-based privacy auditing, are unreliable and fail to detect major leaks. Our findings indicate that even when layered, current heuristic protections are insufficient to adequately protect user data in LLM-based analysis systems.
Paper Structure (21 sections, 1 equation, 7 figures, 10 tables, 4 algorithms)

This paper contains 21 sections, 1 equation, 7 figures, 10 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Background: Clio
  3. Methodology
  4. Threat Model
  5. Cliopatra
  6. Fast Clio Attack
  7. Experimental Evaluation
  8. Experimental Setup
  9. Attacking Clio
  10. Varying Hyperparameters
  11. Stealthy Cliopatra
  12. Potential Mitigations
  13. Related Work
  14. Conclusion
  15. Clio Algorithms and Prompts
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Clio tamkin2024clio's system diagram.
  • Figure 2: Overview of Cliopatra. Public information and the target private information are highlighted in green and red, respectively.
  • Figure 3: Attack success rate (%) for increasing number of symptoms known to the Cliopatra adversary against Clio configured with different LLMs.
  • Figure 4: Attack success rate (%) for Cliopatra with five known symptoms when Clio is configured with a varying number of chats and LLMs.
  • Figure : Clio
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition E.1: Differential Privacy (DP) dwork2006calibrating