Unlocking the diagnostic potential of electrocardiograms through information transfer from cardiac magnetic resonance imaging

TL;DR

The paper addresses the limited diagnostic ability of ECG for comprehensive cardiovascular assessment and the cost of CMR imaging. It introduces a self-supervised MMCL framework that first performs masked data modelling on ECG with a Transformer backbone and then applies multimodal contrastive learning with paired ECG-CMR data to transfer morphology information to ECG; the framework uses a shared latent space of dimension $128$, a masking ratio $\rho=0.8$, and a CLIP-style temperature $\tau$. On 40,044 UK Biobank subjects, the method improves subject-specific disease risk prediction for CAD, AF, and DM and enhances prediction of 61 CMR-derived cardiac phenotypes from ECG compared with unimodal baselines. It also provides a patch-based interpretability module and is open-sourced, signaling broad practical impact for affordable, holistic cardiac screening.

Abstract

Cardiovascular diseases (CVD) can be diagnosed using various diagnostic modalities. The electrocardiogram (ECG) is a cost-effective and widely available diagnostic aid that provides functional information of the heart. However, its ability to classify and spatially localise CVD is limited. In contrast, cardiac magnetic resonance (CMR) imaging provides detailed structural information of the heart and thus enables evidence-based diagnosis of CVD, but long scan times and high costs limit its use in clinical routine. In this work, we present a deep learning strategy for cost-effective and comprehensive cardiac screening solely from ECG. Our approach combines multimodal contrastive learning with masked data modelling to transfer domain-specific information from CMR imaging to ECG representations. In extensive experiments using data from 40,044 UK Biobank subjects, we demonstrate the utility and generalisability of our method for subject-specific risk prediction of CVD and the prediction of cardiac phenotypes using only ECG data. Specifically, our novel multimodal pre-training paradigm improves performance by up to 12.19 % for risk prediction and 27.59 % for phenotype prediction. In a qualitative analysis, we demonstrate that our learned ECG representations incorporate information from CMR image regions of interest. Our entire pipeline is publicly available at https://github.com/oetu/MMCL-ECG-CMR.

Figures (6)

• Figure 1: Overview of our training stages and inference process. (a) Our proposed approach uses masked data modelling to learn meaningful ECG representations, eliminating redundancy inherent to standard $12$-lead ECG. (b) We introduce multimodal contrastive learning to transfer domain-specific information from CMR imaging to ECG. (c) Once pre-trained, the signal encoder is fine-tuned and can be used during inference to predict the risk of cardiovascular diseases and to predict cardiac phenotypes solely from ECG.
• Figure 2: We use masked data modelling to eliminate redundancy inherent to 12-lead ECG, thus generating meaningful ECG representations. To this end, we split the ECG data into patches of predefined size, out of which a random set is masked out. Note that for visualisation purposes the patch size is set to cover a single heartbeat, however, the actual patch size may vary. Only the small subset of visible patches is encoded by the signal encoder. The full set of encoded and masked patches is reconstructed by the decoder.
• Figure 3: We introduce multimodal contrastive learning that combines 12-lead ECG and CMR imaging, enabling self-supervised information transfer from CMR imaging to ECG. ECG and CMR images are embedded separately using unimodal encoders. The representations are projected separately onto a shared latent space, where information is exchanged between both modalities. A passive interpretability module visualises the similarity between the global ECG representation and local CMR image representations, allowing for a qualitative evaluation of the information transfer.
• Figure 4: We use our multimodally pre-trained model to predict 61 cardiac imaging phenotypes solely from ECG data. The graphs show 20 imaging phenotypes of 500 subjects, as well as the linear regression line for the whole test set population. Pearson's correlation coefficient (r) is reported.
• Figure 5: Performance of our multimodal approach with different number of fine-tune training samples. We compare our solution to supervised and self-supervised baseline models. Shaded regions indicate 95 % confidence intervals. Multimodal contrastive learning with masked data modelling generally outperforms all other models at all data quantities and is well suited for predicting risks of rare diseases.