Retrospective Comparative Analysis of Prostate Cancer In-Basket Messages: Responses from Closed-Domain LLM vs. Clinical Teams

TL;DR

Employing RadOnc-GPT for in-basket message draft generation has the potential to alleviate the workload of clinical care teams and reduce healthcare costs by producing high-quality, timely responses.

Abstract

In-basket message interactions play a crucial role in physician-patient communication, occurring during all phases (pre-, during, and post) of a patient's care journey. However, responding to these patients' inquiries has become a significant burden on healthcare workflows, consuming considerable time for clinical care teams. To address this, we introduce RadOnc-GPT, a specialized Large Language Model (LLM) powered by GPT-4 that has been designed with a focus on radiotherapeutic treatment of prostate cancer with advanced prompt engineering, and specifically designed to assist in generating responses. We integrated RadOnc-GPT with patient electronic health records (EHR) from both the hospital-wide EHR database and an internal, radiation-oncology-specific database. RadOnc-GPT was evaluated on 158 previously recorded in-basket message interactions. Quantitative natural language processing (NLP) analysis and two grading studies with clinicians and nurses were used to assess RadOnc-GPT's responses. Our findings indicate that RadOnc-GPT slightly outperformed the clinical care team in "Clarity" and "Empathy," while achieving comparable scores in "Completeness" and "Correctness." RadOnc-GPT is estimated to save 5.2 minutes per message for nurses and 2.4 minutes for clinicians, from reading the inquiry to sending the response. Employing RadOnc-GPT for in-basket message draft generation has the potential to alleviate the workload of clinical care teams and reduce healthcare costs by producing high-quality, timely responses.

Figures (14)

• Figure 1: In-Basket Comparison Study Workflow Overview. From the in-basket messages dataset, we extracted prostate cancer patient inquiries and their corresponding care team responses. RadOnc-GPT, integrated with patients' EHR profiles, generates responses to these inquiries. A randomized dataset containing both RadOnc-GPT and clinical care team responses is then created for NLP-based quantitative evaluation and single-blinded grading by clinical professionals. Clinician and nurse graders can optionally review and match specific responses to patient EHRs using the patient ID.
• Figure 2: Nine Categories of In-Basket Message Patients' Inquiries
• Figure 3: Sentiment Analysis. (A) TextBlob Sentiment Distribution; (B) VADER Sentiment Distribution; (C) NLI Distributions between GPT and Care Team Responses; (D) Semantic Similarity Scores
• Figure 4: Readability Scores and Word and Sentence Counts Comparison.
• Figure 5: Average Score Across all Four Categories