ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography

TL;DR

This work investigates domain adaptation of open-weight LLMs for electrocardiography, comparing finetuning and retrieval-augmented generation against a representative proprietary model. Using a multi-layered evaluation framework that includes MCQ, text similarity, LLM-as-a-judge, and human expert assessments, the study shows that finetuned Llama 3.1 70B excels on in-distribution tasks and approaches the performance of Claude Sonnet 3.7 and RAG in other settings. The results reveal substantial heterogeneity across evaluation modalities, underscoring the need for diverse, statistically robust assessments in domain-specific LLMs. Overall, domain adaptation through finetuning and RAG yields competitive performance with proprietary models, supporting privacy-preserving, locally deployable clinical AI solutions for ECG interpretation and analysis.

Abstract

Domain-adapted open-weight large language models (LLMs) offer promising healthcare applications, from queryable knowledge bases to multimodal assistants, with the crucial advantage of local deployment for privacy preservation. However, optimal adaptation strategies, evaluation methodologies, and performance relative to general-purpose LLMs remain poorly characterized. We investigated these questions in electrocardiography, an important area of cardiovascular medicine, by finetuning open-weight models on domain-specific literature and implementing a multi-layered evaluation framework comparing finetuned models, retrieval-augmented generation (RAG), and Claude Sonnet 3.7 as a representative general-purpose model. Finetuned Llama 3.1 70B achieved superior performance on multiple-choice evaluations and automatic text metrics, ranking second to Claude 3.7 in LLM-as-a-judge assessments. Human expert evaluation favored Claude 3.7 and RAG approaches for complex queries. Finetuned models significantly outperformed their base counterparts across nearly all evaluation modes. Our findings reveal substantial performance heterogeneity across evaluation methodologies, underscoring assessment complexity. Nevertheless, domain-specific adaptation through finetuning and RAG achieves competitive performance with proprietary models, supporting the viability of privacy-preserving, locally deployable clinical solutions.

Figures (8)

• Figure 1: Schematic overview of the core methodology of this study: We study finetuning and retrieval augmented generation as two possible paths towards domain specialization of Llama 3 open-weight LLMs in the domain of electrocardiography. Model performance is assessed in a comprehensive evaluation procedure covering four different categories comparing domain-specialized models in comparison to the respective base models and a commercial general-purpose LLM.
• Figure 2: Correct answers resulting from 417 questions evaluated with LLM-as-a-judge. The number in parentheses after the model name indicates the rank in the statistically robust model ranking. Domain-adapted models explored in this work are highlighted in bold face. Claude Sonnet 3.7 performs best, closely followed by Llama 3.1 70B models. RAG = retrieval-augmented generation; FT = finetuning.
• Figure 3: Human evaluation of the ten factual questions. The number in parentheses after the model name indicates the rank in the statistically robust model ranking. Domain-adapted models explored in this work are highlighted in bold face. Both RAG models perform on par with the top-performing Claude 3.7 Sonnet. RAG = retrieval-augmented generation; FT = finetuning.
• Figure 4: Human evaluation of the 40 semantically complex questions. The number in parentheses after the model name indicates the rank in the statistically robust model ranking. Both RAG models as well as the Llama 3.1 70B base model perform on par with the best-performing Claude 3.7 Sonnet. Domain-adapted models explored in this work are highlighted in bold face. RAG = retrieval-augmented generation; FT = finetuning.
• Figure 5: Evaluation layers in the evaluation approach