IntelliCare: Improving Healthcare Analysis with Variance-Controlled Patient-Level Knowledge from Large Language Models

TL;DR

IntelliCare tackles the challenge of integrating Large Language Model (LLM) knowledge into electronic health record (EHR) predictions while mitigating variance and ambiguity. It achieves this by (1) identifying patient cohorts to supply task-relevant context, (2) generating multiple LLM analyses, and (3) refining knowledge via encoder-driven rectification and perplexity-guided weighting before fusion with an EHR model. The approach yields consistent improvements across three clinical tasks on MIMIC-III and MIMIC-IV and demonstrates robustness to different LLMs and analysis counts, with enhanced stability over baselines. This work offers a practical pathway to incorporate patient-level external knowledge into healthcare decision support, with careful attention to privacy, interpretability, and bias considerations.

Abstract

While pioneering deep learning methods have made great strides in analyzing electronic health record (EHR) data, they often struggle to fully capture the semantics of diverse medical codes from limited data. The integration of external knowledge from Large Language Models (LLMs) presents a promising avenue for improving healthcare predictions. However, LLM analyses may exhibit significant variance due to ambiguity problems and inconsistency issues, hindering their effective utilization. To address these challenges, we propose IntelliCare, a novel framework that leverages LLMs to provide high-quality patient-level external knowledge and enhance existing EHR models. Concretely, IntelliCare identifies patient cohorts and employs task-relevant statistical information to augment LLM understanding and generation, effectively mitigating the ambiguity problem. Additionally, it refines LLM-derived knowledge through a hybrid approach, generating multiple analyses and calibrating them using both the EHR model and perplexity measures. Experimental evaluations on three clinical prediction tasks across two large-scale EHR datasets demonstrate that IntelliCare delivers significant performance improvements to existing methods, highlighting its potential in advancing personalized healthcare predictions and decision support systems.

Figures (4)

• Figure 1: By combining with patient information, LLMs can provide the most relevant knowledge to the patient, rather than listing all information related to a certain medical code, although it might be irrelevant. Thus, employing analysis of patients as external knowledge with LLM avoids the overhead of filtering knowledge. Notably, LLM can provide unique insights in conjunction with clinical tasks and deliver more accurate external knowledge.
• Figure 2: Overview of IntelliCare. Given the electronic medical records of a patient, IntelliCare constructs prompt via the records and task-relevant information within the cohorts to which the patient may belong. We then generate multiple analyses from LLMs for the patient and design a hybrid analysis refinement to calibrate this knowledge. Finally, we combine the refined knowledge with the existing trained EHR model to improve their prediction performance.
• Figure 3: Ablation comparison of IntelliCare for different numbers of analyses on the MIMIC-III dataset.
• Figure 4: Ablation comparison of IntelliCare for different numbers of analyses on the MIMIC-IV dataset.