CARDIUM: Congenital Anomaly Recognition with Diagnostic Images and Unified Medical records

TL;DR

This paper tackles prenatal CHD detection under data scarcity and privacy constraints by introducing CARDIUM, the first public multimodal dataset that combines fetal ultrasound/echocardiographic images with maternal clinical records. It proposes a novel CARDIUM multimodal transformer with dual cross-attention that fuses image and tabular features, yielding a CHD F1 of $0.798 \pm 0.048$ and AUC of $0.974$, outperforming image-only and tabular-only baselines by $11\%$ and $50\%$, respectively. The work demonstrates strong trimester-driven performance, robust unimodal generalization to external datasets, and superior performance against state-of-the-art multimodal methods, while addressing class imbalance via targeted sampling and loss strategies. These results establish CARDIUM as a valuable, openly available benchmark for multimodal prenatal CHD diagnosis and underscore the potential for improving clinical decision support and equitable access to prenatal care.

Abstract

Prenatal diagnosis of Congenital Heart Diseases (CHDs) holds great potential for Artificial Intelligence (AI)-driven solutions. However, collecting high-quality diagnostic data remains difficult due to the rarity of these conditions, resulting in imbalanced and low-quality datasets that hinder model performance. Moreover, no public efforts have been made to integrate multiple sources of information, such as imaging and clinical data, further limiting the ability of AI models to support and enhance clinical decision-making. To overcome these challenges, we introduce the Congenital Anomaly Recognition with Diagnostic Images and Unified Medical records (CARDIUM) dataset, the first publicly available multimodal dataset consolidating fetal ultrasound and echocardiographic images along with maternal clinical records for prenatal CHD detection. Furthermore, we propose a robust multimodal transformer architecture that incorporates a cross-attention mechanism to fuse feature representations from image and tabular data, improving CHD detection by 11% and 50% over image and tabular single-modality approaches, respectively, and achieving an F1 score of 79.8 $\pm$ 4.8% in the CARDIUM dataset. We will publicly release our dataset and code to encourage further research on this unexplored field. Our dataset and code are available at https://github.com/BCV-Uniandes/Cardium, and at the project website https://bcv-uniandes.github.io/CardiumPage/

Figures (5)

• Figure 1: Overview of the CARDIUM dataset. The CARDIUM dataset includes diagnostic images from 1,103 patients and 26 physiological variables from the mother's clinical record.
• Figure 2: CARDIUM dataset statistics. (A) Number of patients with and without CHD (left) and number of images corresponding to patients with and without CHD (right). (B) Trimester distribution: The inner circle represents the overall number of patients with images from the first, second, and third trimesters, while the outer circle distinguishes between positive (darker) and negative (lighter) cases. On the right, we present the percentage of the total dataset corresponding to each gestational period. (C) Distribution of different types of CHDs present in our dataset. (D) Number of patients with and without CHD per fold.
• Figure 3: Overview of the CARDIUM model. We process image and tabular data through modality-specific encoders, $E_I$ and $E_T$, to obtain distinct embeddings. We pass these embeddings through transformer decoder layers, where modality fusion occurs in the cross-attention layer. Then, we concatenate the fused representations and process them through a Multi-Layer Perceptron (MLP) to classify cases as CHD or non-CHD.
• Figure 4: Feature distributions for CHD and non-CHD cases across intermediate outputs of our multimodal model on the CARDIUM dataset. (A) Tabular encoder. (B) Image encoder. (C) Final multimodal module. In plot (C), the point density appears lower compared to plots (A) and (B); however, this lower density is due to overlapping points. Feature distributions are visualized using t-SNE tSNE.
• Figure A: Comparison of multimodal fusion strategies. (A) MLP-Fusion: concatenate modality features, then process them with an MLP. (B) Transformer Encoder Fusion: concatenate features, then process them with a transformer encoder. (C) Transformer Decoder Fusion: process image features with a decoder, then integrate tabular features through cross-attention. (D) Transformer Encoder with Cross-Attention Fusion: each modality is encoded separately, then fused via cross-attention.