Aligning Large Language Models with Healthcare Stakeholders: A Pathway to Trustworthy AI Integration

TL;DR

The paper addresses the challenge of aligning large language models with the knowledge, needs, and values of diverse healthcare stakeholders. It advocates a human-in-the-loop approach across pretraining, instruction tuning, RLHF, and inference-time prompting (including Chain-of-Thought) to embed domain knowledge and safe reasoning into LLMs. It then synthesizes alignment across four stakeholder domains—clinical workflow, patient care, medical education, and payers—illustrating concrete applications such as diagnostic note generation, patient education, exam-preparation support, and EHR-based pre-authorization. The outlook emphasizes multilingual and multimodal expansion, data-efficient alignment, synthetic data, regulatory frameworks, and ongoing human feedback to realize trustworthy, real-world healthcare AI.

Abstract

The wide exploration of large language models (LLMs) raises the awareness of alignment between healthcare stakeholder preferences and model outputs. This alignment becomes a crucial foundation to empower the healthcare workflow effectively, safely, and responsibly. Yet the varying behaviors of LLMs may not always match with healthcare stakeholders' knowledge, demands, and values. To enable a human-AI alignment, healthcare stakeholders will need to perform essential roles in guiding and enhancing the performance of LLMs. Human professionals must participate in the entire life cycle of adopting LLM in healthcare, including training data curation, model training, and inference. In this review, we discuss the approaches, tools, and applications of alignments between healthcare stakeholders and LLMs. We demonstrate that LLMs can better follow human values by properly enhancing healthcare knowledge integration, task understanding, and human guidance. We provide outlooks on enhancing the alignment between humans and LLMs to build trustworthy real-world healthcare applications.

Figures (1)

• Figure 1: Overview of aligning large language models (LLMs) with key healthcare stakeholders and the outlook for trustworthy human–LLM collaboration. We highlight major alignments between healthcare stakeholders and LLMs across various scenarios from clinical workflow, patient care, medical education, and healthcare payers. We also outline future directions for trustworthy human-LLM alignment in healthcare, including improving generalizability, developing cost-efficient alignment, mitigating hallucinations, and establishing regulatory frameworks.