ECG-Reasoning-Benchmark: A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation

Abstract

While Multimodal Large Language Models (MLLMs) show promising performance in automated electrocardiogram interpretation, it remains unclear whether they genuinely perform actual step-by-step reasoning or just rely on superficial visual cues. To investigate this, we introduce \textbf{ECG-Reasoning-Benchmark}, a novel multi-turn evaluation framework comprising over 6,400 samples to systematically assess step-by-step reasoning across 17 core ECG diagnoses. Our comprehensive evaluation of state-of-the-art models reveals a critical failure in executing multi-step logical deduction. Although models possess the medical knowledge to retrieve clinical criteria for a diagnosis, they exhibit near-zero success rates (6% Completion) in maintaining a complete reasoning chain, primarily failing to ground the corresponding ECG findings to the actual visual evidence in the ECG signal. These results demonstrate that current MLLMs bypass actual visual interpretation, exposing a critical flaw in existing training paradigms and underscoring the necessity for robust, reasoning-centric medical AI. The code and data are available at https://github.com/Jwoo5/ecg-reasoning-benchmark.

Figures (21)

• Figure 1: End-to-end schematic of the Automated ECG Analysis Pipeline. The workflow illustrates a systematic progression: from the initial delineation of fundamental waveforms in the raw signal to the quantification of physiological features, which are then mapped into discrete clinical findings and ultimately resulting in a definitive diagnostic conclusion.
• Figure 2: An illustrative example of the 4-step recursive evaluation loop for diagnosing Complete Left Bundle Branch Block. The process begins with an Initial Diagnostic Question, followed by the verification of necessary findings. The figure depicts the first loop, which verifies the presence of "Prolonged QRS duration" grounded in the wave segment and measurement range, and the fourth loop, which confirms the "Notched R wave in lateral leads" with specific lead and wave grounding.
• Figure 3: Comparison of P wave detection before (top) and after (bottom) post-processing. Shaded backgrounds indicate GT annotations, while colored signals show predictions. Our pipeline correctly identifies the unannotated P waves (circled), which are penalized as False Positives due to the missing GT.
• Figure 4: Prevalence of diagnostic labels in the PTB-XL (left) and MIMIC-IV-ECG (right) datasets. Diagnoses directly covered by the 17 core logic diagrams of ECG-Reasoning-Benchmark are highlighted in coral, while conditions that can be indirectly derived from these core diagnoses are highlighted in light coral.
• Figure 5: System prompt provided to the evaluated models