HeartLLM: Discretized ECG Tokenization for LLM-Based Diagnostic Reasoning

TL;DR

HeartLLM introduces a discretized ECG tokenizer that enables LLM-based diagnostic reasoning over 12-lead ECGs by converting continuous waveforms into a shared ECG vocabulary. Through a lead-wise encoder, fixed-scale quantization, autoregressive pretraining, and lightweight LoRA-based instruction tuning, the approach achieves state-of-the-art results on ECG-QA and ECG-Report while demonstrating strong zero-shot generalization. By avoiding explicit cross-modal alignment and leveraging a token-level interface, HeartLLM provides a scalable pathway to open-ended clinical reasoning with ECG data. This framework holds practical potential for integrating LLM-driven ECG interpretation into clinical workflows and broadening task adaptability across datasets and conditions.

Abstract

Electrocardiography (ECG) plays a central role in cardiovascular diagnostics, yet existing automated approaches often struggle to generalize across clinical tasks and offer limited support for open-ended reasoning. We present HeartLLM, a novel framework that integrates time-series (TS) and language modeling by enabling large language models (LLMs) to process 12-lead ECG signals for clinical text generation tasks. Our approach discretizes continuous ECG embeddings into quantized codes using a lead-wise encoder and quantization module. These quantized codes are then mapped to an extended ECG vocabulary to form ECG tokens, enabling the model to process both ECG and natural language inputs within a unified framework. To bridge the modality gap, we pretrain the model on an autoregressive ECG token forecasting task, allowing the LLM to capture temporal dynamics through its inherent language modeling capability. Finally, we perform instruction tuning on both ECG question answering and diagnostic report generation. Without modifying the core model, HeartLLM achieves strong performance across tasks while maintaining generalization to out-of-distribution settings. Extensive experiments demonstrate the effectiveness of each component and highlight the potential of integrating discretized ECG tokens into LLMs for medical reasoning.

Figures (4)

• Figure 1: UMAP visualizations of ECG and text embedding distributions across different models. (a-b) TEST and TIMELLM exhibit clear modality separation. (c) CLIP shows partially aligned clusters, but still fragmented modality boundaries. (d) HeartLLM aligns all ECG representations with text in a shared semantic space, indicating effective modality unification without explicit contrastive pairing.
• Figure 2: Overview of the HeartLLM framework. The model consists of three stages: (1) ECG Tokenizer, where 12-lead ECG signals are encoded by 12 lead-wise encoders and discretized with FSQ into symbolic ECG tokens; (2) ECG Token Pretraining, where an LLM is autoregressively pretrained on ECG tokens to jointly optimize the extended ECG vocabulary and the model using teacher-forcing next-token prediction; and (3) QA/Report Tuning, where the pretrained model is adapted to downstream tasks using lightweight LoRA tuning. Prompts include structured tabular features and textual instructions to guide generation or question answering.
• Figure 3: Ablation study on the PTB-XL. (a) Report generation evaluated by BLEU, METEOR, and ROUGE. (b–d) EM accuracy on QA-Verify, QA-Choose, and QA-Query tasks. We compare the full model (Ours) against three variants: w/o TAB, w/o FT, and w/o DISC.
• Figure 4: Visualization of latent alignment and signal reconstruction. (a-b) show attention maps for clinical queries, with shaded areas denoting high-impact segments. (c) shows reconstructed waveforms from symbolic tokens, demonstrating that discretization suppresses noise overfitting while preserving key information.