Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering

Shaowei Guan, Yu Zhai, Hin Chi Kwok, Jiawei Du, Xinyu Feng, Jing Li, Harry Qin, Vivian Hui

Abstract

Recent advances in Retrieval-Augmented Generation (RAG) have enabled large language models (LLMs) to ground outputs in clinical evidence. However, connecting LLMs with external databases introduces the risk of contextual leakage: a subtle privacy threat where unique combinations of medical details enable patient re-identification even without explicit identifiers. Current benchmarks in healthcare heavily focus on accuracy, ignoring such privacy issues, despite strict regulations like Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR). To fill this gap, we present MedPriv-Bench, the first benchmark specifically designed to jointly evaluate privacy preservation and clinical utility in medical open-ended question answering. Our framework utilizes a multi-agent, human-in-the-loop pipeline to synthesize sensitive medical contexts and clinically relevant queries that create realistic privacy pressure. We establish a standardized evaluation protocol leveraging a pre-trained RoBERTa-Natural Language Inference (NLI) model as an automated judge to quantify data leakage, achieving an average of 85.9% alignment with human experts. Through an extensive evaluation of 9 representative LLMs, we demonstrate a pervasive privacy-utility trade-off. Our findings underscore the necessity of domain-specific benchmarks to validate the safety and efficacy of medical AI systems in privacy-sensitive environments.

MedPriv-Bench: Benchmarking the Privacy-Utility Trade-off of Large Language Models in Medical Open-End Question Answering

Abstract

Recent advances in Retrieval-Augmented Generation (RAG) have enabled large language models (LLMs) to ground outputs in clinical evidence. However, connecting LLMs with external databases introduces the risk of contextual leakage: a subtle privacy threat where unique combinations of medical details enable patient re-identification even without explicit identifiers. Current benchmarks in healthcare heavily focus on accuracy, ignoring such privacy issues, despite strict regulations like Health Insurance Portability and Accountability Act (HIPAA) and General Data Protection Regulation (GDPR). To fill this gap, we present MedPriv-Bench, the first benchmark specifically designed to jointly evaluate privacy preservation and clinical utility in medical open-ended question answering. Our framework utilizes a multi-agent, human-in-the-loop pipeline to synthesize sensitive medical contexts and clinically relevant queries that create realistic privacy pressure. We establish a standardized evaluation protocol leveraging a pre-trained RoBERTa-Natural Language Inference (NLI) model as an automated judge to quantify data leakage, achieving an average of 85.9% alignment with human experts. Through an extensive evaluation of 9 representative LLMs, we demonstrate a pervasive privacy-utility trade-off. Our findings underscore the necessity of domain-specific benchmarks to validate the safety and efficacy of medical AI systems in privacy-sensitive environments.
Paper Structure (45 sections, 9 equations, 5 figures, 7 tables)

This paper contains 45 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical QA Benchmarks.
  4. Open-Ended Evaluation.
  5. Methodology
  6. Data Source
  7. Problem Formulation
  8. Multi-Agent Generation Pipeline
  9. Phase 1: PHI Injection
  10. Phase 2: Question Generation (Threat Model)
  11. Phase 3: Answer Generation
  12. Auto-Validation and Filtering
  13. Evaluation
  14. Evaluation Metrics
  15. Response Utility (0--5).
  16. ...and 30 more sections

Figures (5)

  • Figure 1: Illustration of contextual leakage in healthcare retrieval-augmented generation (RAG) systems.
  • Figure 2: Overview of the four-phase benchmark construction pipeline: (1) Phase 1 uses an Injection Agent to embed contextually coherent PHIs into clinical notes based on a predefined taxonomy. (2) Phase 2 employs a White-box Attack strategy, where a Question Agent crafts queries relying on the injected PHIs. (3) Phase 3 feeds the sensitive notes and queries to an Answer Agent for response generation. (4) Phase 4 applies automated quality filtering and human validation to finalize the dataset.
  • Figure 3: Privacy-Utility Trade-off analysis. Models in the top-right corner represent the ideal balance (High Accuracy, Low Leakage).
  • Figure 4: Impact of Context Volume and Word Constraints on Privacy vs. Utility.
  • Figure 5: Distribution of question types across the benchmark.