Transfer Learning with Clinical Concept Embeddings from Large Language Models

TL;DR

This study investigates how semantic clinical concept embeddings from LLMs affect local and cross-site transfer learning for influenza detection in EHRs across two hospitals. It compares One-Hot baselines, BERT-family embeddings (BioBERT, Med-BERT) and OpenAI embeddings under local and transfer settings using a 1-D CNN classifier, with careful attention to fine-tuning strategies. The findings indicate domain-specific embeddings like Med-BERT yield the best local performance and transfer portability, while generic embeddings require more tuning, and excessive fine-tuning can reduce effectiveness. The work offers practical guidance on selecting embeddings and tuning strategies to enable robust knowledge sharing across healthcare sites.

Abstract

Knowledge sharing is crucial in healthcare, especially when leveraging data from multiple clinical sites to address data scarcity, reduce costs, and enable timely interventions. Transfer learning can facilitate cross-site knowledge transfer, but a major challenge is heterogeneity in clinical concepts across different sites. Large Language Models (LLMs) show significant potential of capturing the semantic meaning of clinical concepts and reducing heterogeneity. This study analyzed electronic health records from two large healthcare systems to assess the impact of semantic embeddings from LLMs on local, shared, and transfer learning models. Results indicate that domain-specific LLMs, such as Med-BERT, consistently outperform in local and direct transfer scenarios, while generic models like OpenAI embeddings require fine-tuning for optimal performance. However, excessive tuning of models with biomedical embeddings may reduce effectiveness, emphasizing the need for balance. This study highlights the importance of domain-specific embeddings and careful model tuning for effective knowledge transfer in healthcare.

Figures (4)

• Figure 1: Workflow overview
• Figure 2: How embeddings, filters, and the max operation summarize patient information: Patient 1 presents two symptoms, and patient 2 presents three. Each symptom is represented by a vector with a length of 2. There are two tasks to detect "body pain" and "fever" respectively. The convolutional layer uses two corresponding filters with the same length of word representative vectors, and captures information related to each task. Blue arrows indicate calculations performed using the "body pain filter." For Patient 1, the "body pain filter" outputs two values: (-6.7620) × (-6) + (-0.6463) × (-1) = 41.2183, and (-0.0534) × (-6) + (0.0267) × (-1) = 0.2937. For Patient 2, the "body pain filter" outputs three values: 0.0286, 27.8794, and 7.3754. The max pooling layer then selects the highest value, assigning 41.2183 as the body pain strength for Patient 1 and 27.8794 for Patient 2.
• Figure 3: Clinical Concepts Embedding CNN Struture
• Figure 4: AUROCs of Local models on AC and SLC