How Much Would a Clinician Edit This Draft? Evaluating LLM Alignment for Patient Message Response Drafting

TL;DR

The paper tackles the problem of aligning LLM-generated drafts for patient portal messages with clinicians, introducing a clinician-informed thematic taxonomy and a two-level EditJudge evaluation to quantify editing workload. It builds an expert-annotated dataset across IPPM, SyPPM, and SoCPPM and evaluates six LLMs with five adaptation strategies, finding substantial epistemic uncertainty in content-level alignment but meaningful gains from theme-driven adaptation. The authors release the thematic framework, evaluation framework, and datasets, demonstrating that personalized, clinician-level adaptation can yield approximately a 25–26% reduction in clinician edits, advancing responsible deployment of AI in patient-clinician communication. The work highlights the need for individualized customization and robust automated evaluation beyond generic prompting to realize reliable AI-assisted message drafting in real-world workflows.

Abstract

Large language models (LLMs) show promise in drafting responses to patient portal messages, yet their integration into clinical workflows raises various concerns, including whether they would actually save clinicians time and effort in their portal workload. We investigate LLM alignment with individual clinicians through a comprehensive evaluation of the patient message response drafting task. We develop a novel taxonomy of thematic elements in clinician responses and propose a novel evaluation framework for assessing clinician editing load of LLM-drafted responses at both content and theme levels. We release an expert-annotated dataset and conduct large-scale evaluations of local and commercial LLMs using various adaptation techniques including thematic prompting, retrieval-augmented generation, supervised fine-tuning, and direct preference optimization. Our results reveal substantial epistemic uncertainty in aligning LLM drafts with clinician responses. While LLMs demonstrate capability in drafting certain thematic elements, they struggle with clinician-aligned generation in other themes, particularly question asking to elicit further information from patients. Theme-driven adaptation strategies yield improvements across most themes. Our findings underscore the necessity of adapting LLMs to individual clinician preferences to enable reliable and responsible use in patient-clinician communication workflows.

Figures (7)

• Figure 1: Patient message response drafting. LLMs draft responses to patient messages, then clinicians edit the draft by deleting and adding content as needed. We evaluate content-level and theme-level alignment between clinicians and LLMs.
• Figure 2: The EditJudge Evaluation Framework for evaluating LLM response drafts. The content-level edit-F1 score identifies matching content in the response draft ($EM$, i.e. true positives), along with expected deletions ($ED$, false positives) and expected additions ($EA$, false negatives) needed in order to align the LLM response draft with the clinician's desired response. The theme-level edit score identifies matching themes, serving as a relaxed evaluation of the theme-level alignment.
• Figure 3: Example de-identified EHR chart summary from our SyPPM patient message response drafting evaluation dataset
• Figure 4: Screenshot of the beginning of a REDCap survey question used to collect clinician responses to patient messages in the SyPPM dataset. The patient's EHR chart and message are first given, then the clinician is prompted with a series of text entry boxes for each response theme described in Section \ref{['sec:data_themes']}.
• Figure 5: Screenshot of the end of a REDCap survey response used to collect clinician responses to patient messages in the SyPPM dataset. After seeing the patient's EHR chart and message, the clinician is prompted with a series of text entry boxes for each response theme described in Section \ref{['sec:data_themes']}. The clinician is also prompted to give any additional thoughts or assumptions they made while drafting their response.