Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation

TL;DR

FetalMind introduces a clinically guided, epistemic-aware vision-language framework for fetal ultrasound interpretation. By coupling Salient Epistemic Disentanglement (SED) with Salient View Preference Optimization (SVPO) and a disease–view bipartite graph, it aligns multi-view reasoning with obstetric workflows. The large-scale FetalSigma-1M dataset enables cross-center training across all gestational stages. Empirical results show significant improvements in both diagnosis and report generation over baselines, validating the value of structured clinical priors for reliable fetal ultrasound AI.

Abstract

Recent medical vision-language models have shown promise on tasks such as VQA, report generation, and anomaly detection. However, most are adapted to structured adult imaging and underperform in fetal ultrasound, which poses challenges of multi-view image reasoning, numerous diseases, and image diversity. To bridge this gap, we introduce FetalMind, a medical AI system tailored to fetal ultrasound for both report generation and diagnosis. Guided by clinical workflow, we propose Salient Epistemic Disentanglement (SED), which injects an expert-curated bipartite graph into the model to decouple view-disease associations and to steer preference selection along clinically faithful steps via reinforcement learning. This design mitigates variability across diseases and heterogeneity across views, reducing learning bottlenecks while aligning the model's inference with obstetric practice. To train FetalMind at scale, we curate FetalSigma-1M dataset, the first large-scale fetal ultrasound report corpus, comprising 20K reports from twelve medical centers, addressing the scarcity of domain data. Extensive experiments show that FetalMind outperforms open- and closed-source baselines across all gestational stages, achieving +14% average gains and +61.2% higher accuracy on critical conditions while remaining efficient, stable, and scalable. Project Page: https://hexiao0275.github.io/FetalMind.

Figures (14)

• Figure 1: (a) Fetal ultrasound workflow. (b) Limitations of vanilla MLLMs on multi-view scans: ❶ A severe imbalance, with abundant visual tokens but limited textual supervision, induces representation collapse; ❷ Fetal imaging spans $>300$ fine-grained diseases, markedly complicating robust diagnosis. (c) FetalMind aligns with the clinical workflow: view examination, abnormality detection, and disease tracing via knowledge.
• Figure 2: Left: Positive correlation ($>0.3$) between diagnostic accuracy and the relative attention advantage of disease-related over non-disease views. Attention is measured by MeanALLQ, defined as the mean attention weight over all query tokens across layers and heads, and results are shown for Qwen-VL 2.5. Right: Multi-center evaluation of report generation and diagnosis with trimester-level diagnostic performance comparison.
• Figure 3: (a) FetalMind aligns with clinical cognition by classifying images into pregnancy-specific views, encoding disease–view keywords as special tokens, and reinforcing their intrinsic associations via salient epistemic disentanglement (SED). (b) FetalSigma-1M comprises 1 million fetal image–report–diagnosis triplets in 12 centers. (c) Overview of SED. Salient views are identified from disease–view graphs (see \ref{['sec:graph']}) and treated as perturbation variables, swapped across fetuses with disease replaced. Bottom-left: intersection- and union-based substitution between diseased regions and views. Bottom-right: SVPO not only injects disease–view knowledge graphs into MLLMs but also enhances inter-disease discriminability
• Figure 4: Illustration of FetalMind versus GPT-5 on a representative case (ID: 12388). The ground-truth diagnosis is a ventricular septal defect (VSD). GPT-5 misclassified the case as normal, likely due to its limited utilization of 2D and Doppler signals. In contrast, FetalMind correctly identified the VSD by integrating multi-view structural cues with blood-flow features. The report is truncated for brevity.
• Figure 5: Diagnostic performance comparison in nine major malformations