Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messages

Joseph Gatto, Parker Seegmiller, Timothy E. Burdick, Sarah Masud Preum

TL;DR

This study introduces an LLM-powered framework for configurable and realistic patient portal message generation which leverages few-shot grounded text generation, requiring only a small number of de-identified patient portal messages to help LLMs better match the true style and tone of real data.

Abstract

Since the COVID-19 pandemic, clinicians have seen a large and sustained influx in patient portal messages, significantly contributing to clinician burnout. To the best of our knowledge, there are no large-scale public patient portal messages corpora researchers can use to build tools to optimize clinician portal workflows. Informed by our ongoing work with a regional hospital, this study introduces an LLM-powered framework for configurable and realistic patient portal message generation. Our approach leverages few-shot grounded text generation, requiring only a small number of de-identified patient portal messages to help LLMs better match the true style and tone of real data. Clinical experts in our team deem this framework as HIPAA-friendly, unlike existing privacy-preserving approaches to synthetic text generation which cannot guarantee all sensitive attributes will be protected. Through extensive quantitative and human evaluation, we show that our framework produces data of higher quality than comparable generation methods as well as all related datasets. We believe this work provides a path forward for (i) the release of large-scale synthetic patient message datasets that are stylistically similar to ground-truth samples and (ii) HIPAA-friendly data generation which requires minimal human de-identification efforts.

In-Context Learning for Preserving Patient Privacy: A Framework for Synthesizing Realistic Patient Portal Messages

TL;DR

This study introduces an LLM-powered framework for configurable and realistic patient portal message generation which leverages few-shot grounded text generation, requiring only a small number of de-identified patient portal messages to help LLMs better match the true style and tone of real data.

Abstract

Since the COVID-19 pandemic, clinicians have seen a large and sustained influx in patient portal messages, significantly contributing to clinician burnout. To the best of our knowledge, there are no large-scale public patient portal messages corpora researchers can use to build tools to optimize clinician portal workflows. Informed by our ongoing work with a regional hospital, this study introduces an LLM-powered framework for configurable and realistic patient portal message generation. Our approach leverages few-shot grounded text generation, requiring only a small number of de-identified patient portal messages to help LLMs better match the true style and tone of real data. Clinical experts in our team deem this framework as HIPAA-friendly, unlike existing privacy-preserving approaches to synthetic text generation which cannot guarantee all sensitive attributes will be protected. Through extensive quantitative and human evaluation, we show that our framework produces data of higher quality than comparable generation methods as well as all related datasets. We believe this work provides a path forward for (i) the release of large-scale synthetic patient message datasets that are stylistically similar to ground-truth samples and (ii) HIPAA-friendly data generation which requires minimal human de-identification efforts.

Paper Structure

This paper contains 23 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Dataset Overview
  3. Methods
  4. Experimental Setup
  5. Evaluation
  6. Results
  7. Discussion, Limitations, and Future Work
  8. Example Data
  9. Comparing Patient Portal Messages to Online Medical Q&A Data
  10. Example Model Outputs
  11. Prompt:
  12. GPT-3.5 Generation
  13. Mixtral 8x7b Generation
  14. PortalGen Generation
  15. Qualitative Analysis:
  16. ...and 8 more sections

Figures (1)

  • Figure 1: Our experiments show that PortalGen produces data which achieves the best balance of perplexity and semantic similarity amongst HIPAA-friendly generation methods. Note that all metrics are scaled between 0-1 (higher=better) for ease of comparison.