Large Language Models for Patient Comments Multi-Label Classification

TL;DR

This paper addresses the challenge of multi-label text classification of inpatient patient feedback (HCAHPS) to identify actionable topics that influence care quality. It introduces a GPT-4 Turbo-based MLTC pipeline augmented with a Protected Health Information (PHI) Detection Framework to de-identify data before processing. Ten topics are defined from patient journey analysis and annotated over 1,089 comments, with comparisons against TF-IDF+Lin-SVM and PLMs, showing LLM-based methods achieve superior performance. The work reveals meaningful associations between classification topics and HCAHPS responses and ratings, offering a privacy-preserving, scalable tool for healthcare providers to monitor and improve patient experiences.

Abstract

Patient experience and care quality are crucial for a hospital's sustainability and reputation. The analysis of patient feedback offers valuable insight into patient satisfaction and outcomes. However, the unstructured nature of these comments poses challenges for traditional machine learning methods following a supervised learning paradigm. This is due to the unavailability of labeled data and the nuances these texts encompass. This research explores leveraging Large Language Models (LLMs) in conducting Multi-label Text Classification (MLTC) of inpatient comments shared after a stay in the hospital. GPT-4 Turbo was leveraged to conduct the classification. However, given the sensitive nature of patients' comments, a security layer is introduced before feeding the data to the LLM through a Protected Health Information (PHI) detection framework, which ensures patients' de-identification. Additionally, using the prompt engineering framework, zero-shot learning, in-context learning, and chain-of-thought prompting were experimented with. Results demonstrate that GPT-4 Turbo, whether following a zero-shot or few-shot setting, outperforms traditional methods and Pre-trained Language Models (PLMs) and achieves the highest overall performance with an F1-score of 76.12% and a weighted F1-score of 73.61% followed closely by the few-shot learning results. Subsequently, the results' association with other patient experience structured variables (e.g., rating) was conducted. The study enhances MLTC through the application of LLMs, offering healthcare practitioners an efficient method to gain deeper insights into patient feedback and deliver prompt, appropriate responses.

Figures (6)

• Figure : Figure 1. Research Problem
• Figure : Figure 2. Research Methodology
• Figure : Figure 3. Patient’s Journey Mapping in the Case of an Elective Surgery
• Figure : Figure 4. Topics Distribution
• Figure : Figure 5. PHI Detection Framework