Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

Corentin Royer, Bjoern Menze, Anjany Sekuboyina

TL;DR

<3-5 sentence high-level summary> MultiMedEval addresses the non-uniform and opaque evaluation of medical vision-language models by delivering an open-source, end-to-end toolkit that standardizes six core tasks (image classification, QA, VQA, report generation, report summarization, and NLI) across 23 datasets and 11 medical modalities. The framework defines task-specific prompts and metrics, provides a simple API for setup and evaluation, and benchmarks notable baselines (RadFM, LLaVA-Med) alongside closed models (MedPaLM M, Maira-1) and BiomedGPT to establish a unified baseline. It highlights the need for consistent evaluation to accelerate fair comparisons and model development in the medical VLM space, and outlines plans to expand tasks, metrics, and integrations with community tools like MONAI and MLCommons. The work aims to foster reproducible benchmarking and community adoption, enabling researchers to focus on model development while ensuring comparable performance assessments across diverse medical domains.

Abstract

We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.

MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Models

TL;DR

<3-5 sentence high-level summary> MultiMedEval addresses the non-uniform and opaque evaluation of medical vision-language models by delivering an open-source, end-to-end toolkit that standardizes six core tasks (image classification, QA, VQA, report generation, report summarization, and NLI) across 23 datasets and 11 medical modalities. The framework defines task-specific prompts and metrics, provides a simple API for setup and evaluation, and benchmarks notable baselines (RadFM, LLaVA-Med) alongside closed models (MedPaLM M, Maira-1) and BiomedGPT to establish a unified baseline. It highlights the need for consistent evaluation to accelerate fair comparisons and model development in the medical VLM space, and outlines plans to expand tasks, metrics, and integrations with community tools like MONAI and MLCommons. The work aims to foster reproducible benchmarking and community adoption, enabling researchers to focus on model development while ensuring comparable performance assessments across diverse medical domains.

Abstract

We introduce MultiMedEval, an open-source toolkit for fair and reproducible evaluation of large, medical vision-language models (VLM). MultiMedEval comprehensively assesses the models' performance on a broad array of six multi-modal tasks, conducted over 23 datasets, and spanning over 11 medical domains. The chosen tasks and performance metrics are based on their widespread adoption in the community and their diversity, ensuring a thorough evaluation of the model's overall generalizability. We open-source a Python toolkit (github.com/corentin-ryr/MultiMedEval) with a simple interface and setup process, enabling the evaluation of any VLM in just a few lines of code. Our goal is to simplify the intricate landscape of VLM evaluation, thus promoting fair and uniform benchmarking of future models.
Paper Structure (16 sections, 1 figure, 11 tables)

This paper contains 16 sections, 1 figure, 11 tables.

Table of Contents

  1. Introduction
  2. Tasks and Evaluations
  3. Multi-class and multi-label image classification.
  4. Question answering.
  5. Visual question answering.
  6. Report generation.
  7. Report summarization.
  8. Natural language inference.
  9. MultiMedEval Setup and Utilization
  10. Parameters.
  11. Batcher.
  12. Evaluation.
  13. Baselines
  14. Conclusion
  15. Future work.
  16. ...and 1 more sections

Figures (1)

  • Figure 1: The left column represents the datasets in MultiMedEval and their size. Each dataset is used in a task, represented in the middle. Finally, we represent the share of each modality in the tool and the modality composition of each task.