Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge

Kang Wang, Chen Qin, Zhang Shi, Haoran Wang, Xiwen Zhang, Chen Chen, Cheng Ouyang, Chengliang Dai, Yuanhan Mo, Chenchen Dai, Xutong Kuang, Ruizhe Li, Xin Chen, Xiuzheng Yue, Song Tian, Alejandro Mora-Rubio, Kumaradevan Punithakumar, Shizhan Gong, Qi Dou, Sina Amirrajab, Yasmina Al Khalil, Cian M. Scannell, Lexiaozi Fan, Huili Yang, Xiaowu Sun, Rob van der Geest, Tewodros Weldebirhan Arega, Fabrice Meriaudeau, Caner Özer, Amin Ranem, John Kalkhof, İlkay Öksüz, Anirban Mukhopadhyay, Abdul Qayyum, Moona Mazher, Steven A Niederer, Carles Garcia-Cabrera, Eric Arazo, Michal K. Grzeszczyk, Szymon Płotka, Wanqin Ma, Xiaomeng Li, Rongjun Ge, Yongqing Kou, Xinrong Chen, He Wang, Chengyan Wang, Wenjia Bai, Shuo Wang

TL;DR

The CMRxMotion study introduces a public benchmark to evaluate how respiratory motion artifacts affect automated CMR analysis, focusing on automated image quality assessment (IQA) and robust cardiac segmentation (RCS). It designs two tasks, releases a controlled, motion-graded dataset from healthy volunteers, and provides a two-stage evaluation with validation and offline testing, reporting that the top IQA method achieves a Cohen's $\kappa$ of $0.631$ while RCS achieves high DSC across structures but degrades with motion, especially for the MYO and RV. The findings demonstrate that IQA is feasible but fine-grained motion grading remains challenging, and that segmentation performance, and thus clinical biomarker accuracy, deteriorates as motion artifacts worsen; human experts still outperform AI in severe artifact cases. Overall, the benchmark emphasizes the need for robust, efficient, motion-aware CMR analysis and offers a valuable resource for ongoing development and validation in this clinically important area.

Abstract

Deep learning models have achieved state-of-the-art performance in automated Cardiac Magnetic Resonance (CMR) analysis. However, the efficacy of these models is highly dependent on the availability of high-quality, artifact-free images. In clinical practice, CMR acquisitions are frequently degraded by respiratory motion, yet the robustness of deep learning models against such artifacts remains an underexplored problem. To promote research in this domain, we organized the MICCAI CMRxMotion challenge. We curated and publicly released a dataset of 320 CMR cine series from 40 healthy volunteers who performed specific breathing protocols to induce a controlled spectrum of motion artifacts. The challenge comprised two tasks: 1) automated image quality assessment to classify images based on motion severity, and 2) robust myocardial segmentation in the presence of motion artifacts. A total of 22 algorithms were submitted and evaluated on the two designated tasks. This paper presents a comprehensive overview of the challenge design and dataset, reports the evaluation results for the top-performing methods, and further investigates the impact of motion artifacts on five clinically relevant biomarkers. All resources and code are publicly available at: https://github.com/CMRxMotion

Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge

TL;DR

The CMRxMotion study introduces a public benchmark to evaluate how respiratory motion artifacts affect automated CMR analysis, focusing on automated image quality assessment (IQA) and robust cardiac segmentation (RCS). It designs two tasks, releases a controlled, motion-graded dataset from healthy volunteers, and provides a two-stage evaluation with validation and offline testing, reporting that the top IQA method achieves a Cohen's of while RCS achieves high DSC across structures but degrades with motion, especially for the MYO and RV. The findings demonstrate that IQA is feasible but fine-grained motion grading remains challenging, and that segmentation performance, and thus clinical biomarker accuracy, deteriorates as motion artifacts worsen; human experts still outperform AI in severe artifact cases. Overall, the benchmark emphasizes the need for robust, efficient, motion-aware CMR analysis and offers a valuable resource for ongoing development and validation in this clinically important area.

Abstract

Deep learning models have achieved state-of-the-art performance in automated Cardiac Magnetic Resonance (CMR) analysis. However, the efficacy of these models is highly dependent on the availability of high-quality, artifact-free images. In clinical practice, CMR acquisitions are frequently degraded by respiratory motion, yet the robustness of deep learning models against such artifacts remains an underexplored problem. To promote research in this domain, we organized the MICCAI CMRxMotion challenge. We curated and publicly released a dataset of 320 CMR cine series from 40 healthy volunteers who performed specific breathing protocols to induce a controlled spectrum of motion artifacts. The challenge comprised two tasks: 1) automated image quality assessment to classify images based on motion severity, and 2) robust myocardial segmentation in the presence of motion artifacts. A total of 22 algorithms were submitted and evaluated on the two designated tasks. This paper presents a comprehensive overview of the challenge design and dataset, reports the evaluation results for the top-performing methods, and further investigates the impact of motion artifacts on five clinically relevant biomarkers. All resources and code are publicly available at: https://github.com/CMRxMotion

Paper Structure

This paper contains 71 sections, 6 equations, 17 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Materials and methods
  4. Data acquisition
  5. Preprocessing
  6. Challenge dataset
  7. Challenge design
  8. Task 1: automated image quality assessment
  9. Motivation
  10. Annotation protocol
  11. Evaluation metrics
  12. Ranking scheme
  13. Task 2: robust cardiac segmentation
  14. Motivation
  15. Annotation protocol
  16. ...and 56 more sections

Figures (17)

  • Figure 1: Visualization example of paired short-axis (SAX) CMR images with different Image Quality Assessment (IQA) scores, acquired from the same volunteer performing four distinct breath-hold behaviors following specific breathing guidance during CMR acquisition. (b) End-diastolic (ED) images, showing three images with progressively increasing IQA scores, are presented in the top row. (c) End-systolic (ES) images, which also display three CMR scans in both axial and coronal planes, are presented in the bottom row.
  • Figure 2: Visualization of segmentation labels for Task 2. (a) Volume rendering of a short-axis CMR image. (b) Volume rendering of the corresponding segmentation mask. (c-h) Paired image slices and segmentation masks for the basal, mid-ventricular, and apical regions. The LV is shown in red, MYO in green, and RV in blue.
  • Figure 3: Confusion matrixes for the IQA task. The horizontal axis represents the predicted labels, and the vertical axis represents the ground truth labels.
  • Figure 4: Nightingale rose chart illustrating the per-class classification performance for the IQA task in terms of (a) Precision and (b) Recall. Each colored group corresponds to a participating team, and the three sectors within each group represent performance on the three IQA labels.
  • Figure 5: Forest plots of IQA performance. The means (depicted as red and blue squares) and the $\bold{95\%}$ confidence intervals (shown as horizontal lines) are estimated from $\bold{10,000}$ bootstrap samples of the image quality assessment task on the test dataset. The x-axis represents the metric values, while the y-axis denotes the participating teams. The horizontal lines indicate the $\bold{95\%}$ confidence intervals ($\bold{95\%}$ CI) derived from the bootstrap analysis. The means (represented by red and blue squares) are calculated by averaging the respective metric across all cases of test set.
  • ...and 12 more figures