Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound

Hussain Alasmawi, Numan Saeed, Mohammad Yaqub

TL;DR

Fetal-Gauge addresses the lack of standardized evaluation for vision-language models in fetal ultrasound by introducing the largest multi-task benchmark that integrates 13 public datasets and phantom data. It defines five clinically relevant MCQ tasks and systematically evaluates 15 state-of-the-art VLMs, revealing a substantial performance gap with GPT-5 achieving only 55% accuracy. The study demonstrates that ultrasound-specific training and domain adaptation markedly impact performance, with LoRA fine-tuning achieving notable gains (e.g., LLama-3.2-11B from 33% to 85%). By establishing a rigorous evaluation framework, Fetal-Gauge provides a foundation for advancing multimodal deep learning in prenatal care and guiding the development of domain-specific approaches to improve global prenatal health outcomes.

Abstract

The growing demand for prenatal ultrasound imaging has intensified a global shortage of trained sonographers, creating barriers to essential fetal health monitoring. Deep learning has the potential to enhance sonographers' efficiency and support the training of new practitioners. Vision-Language Models (VLMs) are particularly promising for ultrasound interpretation, as they can jointly process images and text to perform multiple clinical tasks within a single framework. However, despite the expansion of VLMs, no standardized benchmark exists to evaluate their performance in fetal ultrasound imaging. This gap is primarily due to the modality's challenging nature, operator dependency, and the limited public availability of datasets. To address this gap, we present Fetal-Gauge, the first and largest visual question answering benchmark specifically designed to evaluate VLMs across various fetal ultrasound tasks. Our benchmark comprises over 42,000 images and 93,000 question-answer pairs, spanning anatomical plane identification, visual grounding of anatomical structures, fetal orientation assessment, clinical view conformity, and clinical diagnosis. We systematically evaluate several state-of-the-art VLMs, including general-purpose and medical-specific models, and reveal a substantial performance gap: the best-performing model achieves only 55\% accuracy, far below clinical requirements. Our analysis identifies critical limitations of current VLMs in fetal ultrasound interpretation, highlighting the urgent need for domain-adapted architectures and specialized training approaches. Fetal-Gauge establishes a rigorous foundation for advancing multimodal deep learning in prenatal care and provides a pathway toward addressing global healthcare accessibility challenges. Our benchmark will be publicly available once the paper gets accepted.

FETAL-GAUGE: A Benchmark for Assessing Vision-Language Models in Fetal Ultrasound

TL;DR

Fetal-Gauge addresses the lack of standardized evaluation for vision-language models in fetal ultrasound by introducing the largest multi-task benchmark that integrates 13 public datasets and phantom data. It defines five clinically relevant MCQ tasks and systematically evaluates 15 state-of-the-art VLMs, revealing a substantial performance gap with GPT-5 achieving only 55% accuracy. The study demonstrates that ultrasound-specific training and domain adaptation markedly impact performance, with LoRA fine-tuning achieving notable gains (e.g., LLama-3.2-11B from 33% to 85%). By establishing a rigorous evaluation framework, Fetal-Gauge provides a foundation for advancing multimodal deep learning in prenatal care and guiding the development of domain-specific approaches to improve global prenatal health outcomes.

Abstract

The growing demand for prenatal ultrasound imaging has intensified a global shortage of trained sonographers, creating barriers to essential fetal health monitoring. Deep learning has the potential to enhance sonographers' efficiency and support the training of new practitioners. Vision-Language Models (VLMs) are particularly promising for ultrasound interpretation, as they can jointly process images and text to perform multiple clinical tasks within a single framework. However, despite the expansion of VLMs, no standardized benchmark exists to evaluate their performance in fetal ultrasound imaging. This gap is primarily due to the modality's challenging nature, operator dependency, and the limited public availability of datasets. To address this gap, we present Fetal-Gauge, the first and largest visual question answering benchmark specifically designed to evaluate VLMs across various fetal ultrasound tasks. Our benchmark comprises over 42,000 images and 93,000 question-answer pairs, spanning anatomical plane identification, visual grounding of anatomical structures, fetal orientation assessment, clinical view conformity, and clinical diagnosis. We systematically evaluate several state-of-the-art VLMs, including general-purpose and medical-specific models, and reveal a substantial performance gap: the best-performing model achieves only 55\% accuracy, far below clinical requirements. Our analysis identifies critical limitations of current VLMs in fetal ultrasound interpretation, highlighting the urgent need for domain-adapted architectures and specialized training approaches. Fetal-Gauge establishes a rigorous foundation for advancing multimodal deep learning in prenatal care and provides a pathway toward addressing global healthcare accessibility challenges. Our benchmark will be publicly available once the paper gets accepted.
Paper Structure (20 sections, 5 figures, 6 tables)

This paper contains 20 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Visual Question Answering Datasets
  4. Vision-Language Models in Medical Imaging
  5. The Fetal-Gauge Benchmark
  6. Task Design
  7. Dataset Curation and Standardization
  8. Data partitioning and Statistics
  9. The Role of Phantom Data
  10. Evaluations and Results
  11. Evaluation Setup
  12. Overall Performance Analysis
  13. Task-Specific Performance
  14. Domain Adaptation Through Fine-tuning
  15. Phantom vs. Clinical Performance
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Number of questions categorized by the number of models answering correctly. Each bar represents how many of the 15 models answered a question correctly. For example, the bar at 15 represents the set of questions that all models answered correctly, comprising only 38 correctly answered questions out of a total of 21,468.
  • Figure 2: Question types in the dataset, along with representative sample images for each type, highlighting the diversity of visual content and associated questions.
  • Figure 3: Distribution of benchmark tasks across anatomical regions. Colored segments represent question categories (legend on the right), with proportions shown within each bar and total counts indicated on the right. PI: Anatomical Plane Identification, VG: Visual Grounding of Anatomical Structures, VC: Clinical View Conformity, FO: Fetal Orientation Assessment, CD: Clinical Diagnosis.
  • Figure 4: Bar plot presenting model accuracy on phantom and clinical ultrasound images through two questions ('Plane of the image' and 'Bounding box label')
  • Figure 5: