Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing LLMs' Clinical Reasoning with Real-World Data from a Nationwide Sepsis Registry

Junu Kim, Chaeeun Shim, Sungjin Park, Su Yeon Lee, Gee Young Suh, Chae-Man Lim, Seong Jin Choi, Song Mi Moon, Kyoung-Ho Song, Eu Suk Kim, Hong Bin Kim, Sejoong Kim, Chami Im, Dong-Wan Kang, Yong Soo Kim, Hee-Joon Bae, Sung Yoon Lim, Han-Gil Jeong, Edward Choi

TL;DR

This work tackles the limited clinical reasoning of LLMs in real-world practice by training on real-world clinical data from a nationwide sepsis registry. The authors construct reasoning-intensive, masked-feature denoising questions from patient records and fine-tune Phi-4 with reinforcement learning (GRPO) to create C-Reason. Results show strong in-domain performance, expert preference, and notable cross-cohort, cross-task, open-ended, and cross-disease generalization, including datasets like MIMIC-III, AKI cohorts, and stroke registries. The approach emphasizes scalability and privacy-preserving data use, suggesting future directions toward large-scale, multi-disease clinical data and interactive clinician-LLM collaboration for robust clinical reasoning tools with potential for real-world integration.

Abstract

Although large language models (LLMs) have demonstrated impressive reasoning capabilities across general domains, their effectiveness in real-world clinical practice remains limited. This is likely due to their insufficient exposure to real-world clinical data during training, as such data is typically not included due to privacy concerns. To address this, we propose enhancing the clinical reasoning capabilities of LLMs by leveraging real-world clinical data. We constructed reasoning-intensive questions from a nationwide sepsis registry and fine-tuned Phi-4 on these questions using reinforcement learning, resulting in C-Reason. C-Reason exhibited strong clinical reasoning capabilities on the in-domain test set, as evidenced by both quantitative metrics and expert evaluations. Furthermore, its enhanced reasoning capabilities generalized to a sepsis dataset involving different tasks and patient cohorts, an open-ended consultations on antibiotics use task, and other diseases. Future research should focus on training LLMs with large-scale, multi-disease clinical datasets to develop more powerful, general-purpose clinical reasoning models.

Enhancing LLMs' Clinical Reasoning with Real-World Data from a Nationwide Sepsis Registry

TL;DR

This work tackles the limited clinical reasoning of LLMs in real-world practice by training on real-world clinical data from a nationwide sepsis registry. The authors construct reasoning-intensive, masked-feature denoising questions from patient records and fine-tune Phi-4 with reinforcement learning (GRPO) to create C-Reason. Results show strong in-domain performance, expert preference, and notable cross-cohort, cross-task, open-ended, and cross-disease generalization, including datasets like MIMIC-III, AKI cohorts, and stroke registries. The approach emphasizes scalability and privacy-preserving data use, suggesting future directions toward large-scale, multi-disease clinical data and interactive clinician-LLM collaboration for robust clinical reasoning tools with potential for real-world integration.

Abstract

Although large language models (LLMs) have demonstrated impressive reasoning capabilities across general domains, their effectiveness in real-world clinical practice remains limited. This is likely due to their insufficient exposure to real-world clinical data during training, as such data is typically not included due to privacy concerns. To address this, we propose enhancing the clinical reasoning capabilities of LLMs by leveraging real-world clinical data. We constructed reasoning-intensive questions from a nationwide sepsis registry and fine-tuned Phi-4 on these questions using reinforcement learning, resulting in C-Reason. C-Reason exhibited strong clinical reasoning capabilities on the in-domain test set, as evidenced by both quantitative metrics and expert evaluations. Furthermore, its enhanced reasoning capabilities generalized to a sepsis dataset involving different tasks and patient cohorts, an open-ended consultations on antibiotics use task, and other diseases. Future research should focus on training LLMs with large-scale, multi-disease clinical datasets to develop more powerful, general-purpose clinical reasoning models.
Paper Structure (17 sections, 9 figures, 5 tables)

This paper contains 17 sections, 9 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Results
  5. Clinical Data Training Improves Clinical Reasoning on In-Domain Data
  6. Trained Clinical Reasoning Generalizes Across Cohorts and Tasks
  7. Trained Clinical Reasoning Generalizes to Open-Ended Task
  8. Trained Clinical Reasoning Generalizes Across Diseases
  9. Discussion
  10. Data Statistics
  11. Data Samples and Prompts
  12. Implementation Details
  13. Multiple-Choice Question Generation
  14. Training
  15. Performance Evaluation
  16. ...and 2 more sections

Figures (9)

  • Figure 1: (a) Motivation and Approach. LLMs are primarily trained on web corpora, which leads to insufficient exposure to real-world clinical data and results in limited clinical reasoning capabilities. To address this gap, we further trained an LLM on clinical data, thereby enhancing its real-world clinical reasoning performance. (b) Illustration of the Proposed Method. First, multiple-choice denoising questions are generated from the clinical data (sepsis registry). Then, the LLM generates multiple reasonings for each question and the rewards are calculated based on their correctness. Finally, the model is optimized using the GRPO algorithmshao2024deepseekmath.
  • Figure 2: Reasoning Expert Evaluation Results. We report win rate (%) for each task.
  • Figure 3: Case Analysis - Appropriateness of Initial Empirical Therapy
  • Figure 4: Case Analysis - Consultations on Antibiotics Use
  • Figure 5: Sepsis Registry Expert Evaluation UI
  • ...and 4 more figures