MedVision: Dataset and Benchmark for Quantitative Medical Image Analysis

TL;DR

MedVision tackles the lack of quantitative reasoning in medical vision-language models by introducing a large-scale, multi-modality dataset with rich measurement annotations across 22 public datasets. It defines three clinically relevant tasks—detection, tumor/lesion size estimation, and angle/distance measurement—and provides a rigorous evaluation framework for both open-weight and fine-tuned models. The study shows pretrained VLMs struggle with precise localization and numeric measurements, while supervised fine-tuning with MedVision substantially improves performance, though small-structure detection and some angle/distance tasks remain challenging. By releasing data, checkpoints, and code, MedVision lays a practical foundation for robust quantitative reasoning in medical image analysis and cross-domain generalization.

Abstract

Current vision-language models (VLMs) in medicine are primarily designed for categorical question answering (e.g., "Is this normal or abnormal?") or qualitative descriptive tasks. However, clinical decision-making often relies on quantitative assessments, such as measuring the size of a tumor or the angle of a joint, from which physicians draw their own diagnostic conclusions. This quantitative reasoning capability remains underexplored and poorly supported in existing VLMs. In this work, we introduce MedVision, a large-scale dataset and benchmark specifically designed to evaluate and improve VLMs on quantitative medical image analysis. MedVision spans 22 public datasets covering diverse anatomies and modalities, with 30.8 million image-annotation pairs. We focus on three representative quantitative tasks: (1) detection of anatomical structures and abnormalities, (2) tumor/lesion (T/L) size estimation, and (3) angle/distance (A/D) measurement. Our benchmarks show that current off-the-shelf VLMs perform poorly on these tasks. However, with supervised fine-tuning on MedVision, we significantly enhance their performance across detection, T/L estimation, and A/D measurement, demonstrating reduced error rates and improved precision. This work provides a foundation for developing VLMs with robust quantitative reasoning capabilities in medical imaging. Code and data are available at https://medvision-vlm.github.io.

Figures (15)

• Figure 1: Overview of MedVision dataset construction and benchmark design (left), and dataset summary (right). The MedVision dataset was constructed for out-of-the-box use: load_dataset is all you need. Appendix \ref{['appendix:subsec:dataset_examples']} shows examples of imaging data and quantitative annotation.
• Figure 2: Tumor/lesion size annotation. An ellipse is fitted to the tumor/lesion mask and 4 landmarks are recorded. The largest diameter and its perpendicular diameter are measured.
• Figure 3: Landmarks in the Ceph-Bio-400 (top-left) and FeTA24 datasets. Ground truth angle and distance measurements are calculated from these landmarks.
• Figure 4: SFT improves VLM performance on healthy anatomical structures and tumor/lesion detection tasks, while the latter remains more challenging.
• Figure 5: Detection performance on plane-OOD (left) and target-OOD (right) data. Tumor/lesion (T/L) labels are shown in color. The SFT model shows better generalizability.