Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models

Francesca Filice, Edoardo De Rose, Simone Bartucci, Francesco Calimeri, Simona Perri

TL;DR

The paper addresses the need to evaluate ECG foundation models beyond accuracy, focusing on how embeddings generalize and align with clinical signals. It proposes a representation-aware benchmarking framework that combines $F1$-based performance with SHAP-based feature attribution and UMAP-based embedding analysis, applied to frozen embeddings from 4 ECG-expert FMs across 4 geographically diverse datasets and data-scarcity conditions. Key findings show that ECGFounder and ECG-FM yield more dataset-invariant and clinically separable embeddings, while HuBERT-ECG and ECG-JEPA exhibit stronger dataset dependence and less effective label discrimination, highlighting that larger models do not automatically guarantee better representations. The framework offers practical guidance for safe ECG AI deployment and provides an open-source benchmarking workflow for the research community.

Abstract

The electrocardiogram (ECG) is a cost-effective, highly accessible and widely employed diagnostic tool. With the advent of Foundation Models (FMs), the field of AI-assisted ECG interpretation has begun to evolve, as they enable model reuse across different tasks by relying on embeddings. However, to responsibly employ FMs, it is crucial to rigorously assess to which extent the embeddings they produce are generalizable, particularly in error-sensitive domains such as healthcare. Although prior works have already addressed the problem of benchmarking ECG-expert FMs, they focus predominantly on the evaluation of downstream performance. To fill this gap, this study aims to find an in-depth, comprehensive benchmarking framework for FMs, with a specific focus on ECG-expert ones. To this aim, we introduce a benchmark methodology that complements performance-based evaluation with representation-level analysis, leveraging SHAP and UMAP techniques. Furthermore, we rely on the methodology for carrying out an extensive evaluation of several ECG-expert FMs pretrained via state-of-the-art techniques over different cross-continental datasets and data availability settings; this includes ones featuring data scarcity, a fairly common situation in real-world medical scenarios. Experimental results show that our benchmarking protocol provides a rich insight of ECG-expert FMs' embedded patterns, enabling a deeper understanding of their representational structure and generalizability.

Looking Beyond Accuracy: A Holistic Benchmark of ECG Foundation Models

TL;DR

The paper addresses the need to evaluate ECG foundation models beyond accuracy, focusing on how embeddings generalize and align with clinical signals. It proposes a representation-aware benchmarking framework that combines -based performance with SHAP-based feature attribution and UMAP-based embedding analysis, applied to frozen embeddings from 4 ECG-expert FMs across 4 geographically diverse datasets and data-scarcity conditions. Key findings show that ECGFounder and ECG-FM yield more dataset-invariant and clinically separable embeddings, while HuBERT-ECG and ECG-JEPA exhibit stronger dataset dependence and less effective label discrimination, highlighting that larger models do not automatically guarantee better representations. The framework offers practical guidance for safe ECG AI deployment and provides an open-source benchmarking workflow for the research community.

Abstract

The electrocardiogram (ECG) is a cost-effective, highly accessible and widely employed diagnostic tool. With the advent of Foundation Models (FMs), the field of AI-assisted ECG interpretation has begun to evolve, as they enable model reuse across different tasks by relying on embeddings. However, to responsibly employ FMs, it is crucial to rigorously assess to which extent the embeddings they produce are generalizable, particularly in error-sensitive domains such as healthcare. Although prior works have already addressed the problem of benchmarking ECG-expert FMs, they focus predominantly on the evaluation of downstream performance. To fill this gap, this study aims to find an in-depth, comprehensive benchmarking framework for FMs, with a specific focus on ECG-expert ones. To this aim, we introduce a benchmark methodology that complements performance-based evaluation with representation-level analysis, leveraging SHAP and UMAP techniques. Furthermore, we rely on the methodology for carrying out an extensive evaluation of several ECG-expert FMs pretrained via state-of-the-art techniques over different cross-continental datasets and data availability settings; this includes ones featuring data scarcity, a fairly common situation in real-world medical scenarios. Experimental results show that our benchmarking protocol provides a rich insight of ECG-expert FMs' embedded patterns, enabling a deeper understanding of their representational structure and generalizability.
Paper Structure (21 sections, 2 figures, 4 tables, 1 algorithm)

This paper contains 21 sections, 2 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Datasets
  5. Foundation Models
  6. ECG-FM.
  7. ECGFounder.
  8. HuBERT-ECG.
  9. ECG-JEPA.
  10. Evaluating Performances
  11. Evaluating Embedded Representations
  12. Ranking Feature Importance.
  13. Visualizing and Evaluating the Embedding Space.
  14. Experimental Settings
  15. Reproducibility
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Shared top-50 features rates across datasets. Features are ranked via $C^*_i$ as discussed in Section \ref{['par:shap_feature_ranking']} and described in Algorithm \ref{['alg:shap_umap']}.
  • Figure 2: UMAP-based embedding space visualization across the AF (a-f) and CD (g-l) classes for all the encompassed FMs. Samples are colored according to ground-truth labels. Panels are ordered column-wise.