Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

Srija Mukhopadhyay, Sathwik Reddy, Shruthi Muthukumar, Jisun An, Ponnurangam Kumaraguru

TL;DR

PrivacyBench provides a privacy-centric benchmark to audit contextual privacy in personalized AI by embedding ground-truth secrets within evolving social contexts and evaluating multi-turn interactions. The framework combines a synthetic data-generation pipeline with direct and indirect conversational probes and a standardized set of leakage, over-secrecy, retrieval, and persona-consistency metrics, evaluated via automated judges and human checks. Empirical results reveal substantial baseline leakage in RAG-based systems, with leakage reduced by privacy-aware prompts but not eliminated, and a persistent high Inappropriate Retrieval Rate at the retrieval layer. The work highlights architectural gaps and argues for privacy-by-design safeguards, offering a foundational benchmark and methodology for building trustworthy, socially aware personalized agents.

Abstract

Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone.

PrivacyBench: A Conversational Benchmark for Evaluating Privacy in Personalized AI

TL;DR

PrivacyBench provides a privacy-centric benchmark to audit contextual privacy in personalized AI by embedding ground-truth secrets within evolving social contexts and evaluating multi-turn interactions. The framework combines a synthetic data-generation pipeline with direct and indirect conversational probes and a standardized set of leakage, over-secrecy, retrieval, and persona-consistency metrics, evaluated via automated judges and human checks. Empirical results reveal substantial baseline leakage in RAG-based systems, with leakage reduced by privacy-aware prompts but not eliminated, and a persistent high Inappropriate Retrieval Rate at the retrieval layer. The work highlights architectural gaps and argues for privacy-by-design safeguards, offering a foundational benchmark and methodology for building trustworthy, socially aware personalized agents.

Abstract

Personalized AI agents rely on access to a user's digital footprint, which often includes sensitive data from private emails, chats and purchase histories. Yet this access creates a fundamental societal and privacy risk: systems lacking social-context awareness can unintentionally expose user secrets, threatening digital well-being. We introduce PrivacyBench, a benchmark with socially grounded datasets containing embedded secrets and a multi-turn conversational evaluation to measure secret preservation. Testing Retrieval-Augmented Generation (RAG) assistants reveals that they leak secrets in up to 26.56% of interactions. A privacy-aware prompt lowers leakage to 5.12%, yet this measure offers only partial mitigation. The retrieval mechanism continues to access sensitive data indiscriminately, which shifts the entire burden of privacy preservation onto the generator. This creates a single point of failure, rendering current architectures unsafe for wide-scale deployment. Our findings underscore the urgent need for structural, privacy-by-design safeguards to ensure an ethical and inclusive web for everyone.
Paper Structure (55 sections, 1 figure, 3 tables, 1 algorithm)

This paper contains 55 sections, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Personalization Datasets
  4. Privacy Preservation
  5. Evaluation Metrics
  6. A Privacy-Centric Dataset
  7. Community Simulation
  8. Evolving Social Graph
  9. Dynamic User Profiles
  10. Digital Footprint Generation
  11. Incorporating Secrets
  12. Evaluation Framework
  13. Conversational Probing Strategies
  14. Direct Probing Strategy
  15. Indirect Probing Strategy
  16. ...and 40 more sections

Figures (1)

  • Figure 1: Our pipeline starts with generating seed users and then building a community around them using a LLM. We then follow a two phase process where in the first phase we generate robust profiles and then in the second phase we generate documents to mimic real world interactions to benchmark personalized generation systems with a focus on temporal awareness and privacy considerations.