Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models

Zhiyuan Wang, Fangxu Yuan, Virginia LeBaron, Tabor Flickinger, Laura E. Barnes

TL;DR

This work addresses the need for scalable, nuanced evaluation of patient-provider communication in palliative care by evaluating large language models (LLMs) as evaluators. It demonstrates that GPT-4, with advanced prompting strategies such as Chain-of-Thought, can achieve near-human accuracy in identifying key communication metrics, while an open-source Llama2-13b can be adapted for in-house deployment via parameter-efficient fine-tuning using synthetic data. The study generates 3,000 synthetic dialogue segments with GPT-4 and trains LoRA-tuned Llama2-13b, showing substantial performance gains and highlighting the practical viability of smaller, task-specific LLMs for clinical feedback generation. It discusses ethical, privacy, and real-world integration considerations, outlining a roadmap for stakeholder-centered, multimodal, and clinically validated deployments that could enhance patient outcomes through improved communication.

Abstract

Effective patient-provider communication is crucial in clinical care, directly impacting patient outcomes and quality of life. Traditional evaluation methods, such as human ratings, patient feedback, and provider self-assessments, are often limited by high costs and scalability issues. Although existing natural language processing (NLP) techniques show promise, they struggle with the nuances of clinical communication and require sensitive clinical data for training, reducing their effectiveness in real-world applications. Emerging large language models (LLMs) offer a new approach to assessing complex communication metrics, with the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities. Specifically, using simulated scripts crafted and labeled by healthcare professionals, we test proprietary models (e.g., GPT-4) and fine-tune open-source LLMs (e.g., LLaMA2) with a synthetic dataset generated by GPT-4 to evaluate clinical conversations, to identify key metrics such as `understanding' and `empathy'. Our findings demonstrated LLMs' superior performance in evaluating clinical communication, providing actionable feedback with reasoning, and demonstrating the feasibility and practical viability of developing in-house LLMs. This research highlights LLMs' potential to enhance patient-provider interactions and lays the groundwork for downstream steps in developing LLM-empowered clinical health systems.

PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models

TL;DR

This work addresses the need for scalable, nuanced evaluation of patient-provider communication in palliative care by evaluating large language models (LLMs) as evaluators. It demonstrates that GPT-4, with advanced prompting strategies such as Chain-of-Thought, can achieve near-human accuracy in identifying key communication metrics, while an open-source Llama2-13b can be adapted for in-house deployment via parameter-efficient fine-tuning using synthetic data. The study generates 3,000 synthetic dialogue segments with GPT-4 and trains LoRA-tuned Llama2-13b, showing substantial performance gains and highlighting the practical viability of smaller, task-specific LLMs for clinical feedback generation. It discusses ethical, privacy, and real-world integration considerations, outlining a roadmap for stakeholder-centered, multimodal, and clinically validated deployments that could enhance patient outcomes through improved communication.

Abstract

Effective patient-provider communication is crucial in clinical care, directly impacting patient outcomes and quality of life. Traditional evaluation methods, such as human ratings, patient feedback, and provider self-assessments, are often limited by high costs and scalability issues. Although existing natural language processing (NLP) techniques show promise, they struggle with the nuances of clinical communication and require sensitive clinical data for training, reducing their effectiveness in real-world applications. Emerging large language models (LLMs) offer a new approach to assessing complex communication metrics, with the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities. Specifically, using simulated scripts crafted and labeled by healthcare professionals, we test proprietary models (e.g., GPT-4) and fine-tune open-source LLMs (e.g., LLaMA2) with a synthetic dataset generated by GPT-4 to evaluate clinical conversations, to identify key metrics such as `understanding' and `empathy'. Our findings demonstrated LLMs' superior performance in evaluating clinical communication, providing actionable feedback with reasoning, and demonstrating the feasibility and practical viability of developing in-house LLMs. This research highlights LLMs' potential to enhance patient-provider interactions and lays the groundwork for downstream steps in developing LLM-empowered clinical health systems.
Paper Structure (25 sections, 3 equations, 5 figures, 4 tables)

This paper contains 25 sections, 3 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Clinical Communication Evaluation
  4. Digital Technology in Clinical Communication and Evaluation
  5. Large Language Models in Health Care and Communication
  6. Feasibility Test of LLMs in Identifying Clinical Communication Metrics
  7. Generation of Clinical Communication Benchmark Scripts
  8. Evaluation Approach
  9. Standard Prompt Structure:
  10. Enhanced Reasoning Prompts:
  11. Evaluation Results
  12. Qualitative Notes
  13. Fine-Tuning LLMs To Be Task-Specific for Clinical Communication Evaluation
  14. Generating Synthetic Conversation Evaluation Datasets using GPT-4
  15. Parameter-Efficient Fine-Tuning of Llama2-13b
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Prompt design of patient-provider communication evaluation in palliative care.
  • Figure 2: Prompt design for diversified and realistic synthetic dataset of palliative care clinical communication.
  • Figure 3: Illustrative examples of evaluation outputs generated by GPT-4.
  • Figure 4: Illustrative Dialogues Demonstrating Communication Practices with Standard Generation Prompt$Prompt_{\textbf{generation}}$ Using GPT-4 Generated Synthetic Data.
  • Figure 5: Illustrative Dialogues Demonstrating Communication Practices with CoT-Based Generation Prompt$Prompt_{\textbf{generation\_CoT}}$ Using GPT-4 Generated Synthetic Data.