Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedSG-Bench: A Benchmark for Medical Image Sequences Grounding

Jingkun Yue, Siqi Zhang, Zinan Jia, Huihuan Xu, Zongbo Han, Xiaohong Liu, Guangyu Wang

TL;DR

MedSG-Bench tackles the gap in medical visual grounding by introducing the first benchmark for sequential medical image grounding, covering eight VQA-style tasks across two paradigms and integrating data from 76 datasets and 10 modalities. It demonstrates that current MLLMs struggle with fine-grained grounding in image sequences, motivating the creation of MedSG-188K for instruction tuning and MedSeq-Grounder to advance research in this area. The work provides a comprehensive evaluation framework, a large-scale instructional dataset, and a purpose-built model, aiming to spur progress in reliable, cross-image clinical reasoning. Together, these contributions offer a practical foundation for improving sequential grounding in medical imaging and may enhance clinical decision support and interpretability of multimodal AI systems.

Abstract

Visual grounding is essential for precise perception and reasoning in multimodal large language models (MLLMs), especially in medical imaging domains. While existing medical visual grounding benchmarks primarily focus on single-image scenarios, real-world clinical applications often involve sequential images, where accurate lesion localization across different modalities and temporal tracking of disease progression (e.g., pre- vs. post-treatment comparison) require fine-grained cross-image semantic alignment and context-aware reasoning. To remedy the underrepresentation of image sequences in existing medical visual grounding benchmarks, we propose MedSG-Bench, the first benchmark tailored for Medical Image Sequences Grounding. It comprises eight VQA-style tasks, formulated into two paradigms of the grounding tasks, including 1) Image Difference Grounding, which focuses on detecting change regions across images, and 2) Image Consistency Grounding, which emphasizes detection of consistent or shared semantics across sequential images. MedSG-Bench covers 76 public datasets, 10 medical imaging modalities, and a wide spectrum of anatomical structures and diseases, totaling 9,630 question-answer pairs. We benchmark both general-purpose MLLMs (e.g., Qwen2.5-VL) and medical-domain specialized MLLMs (e.g., HuatuoGPT-vision), observing that even the advanced models exhibit substantial limitations in medical sequential grounding tasks. To advance this field, we construct MedSG-188K, a large-scale instruction-tuning dataset tailored for sequential visual grounding, and further develop MedSeq-Grounder, an MLLM designed to facilitate future research on fine-grained understanding across medical sequential images. The benchmark, dataset, and model are available at https://huggingface.co/MedSG-Bench

MedSG-Bench: A Benchmark for Medical Image Sequences Grounding

TL;DR

MedSG-Bench tackles the gap in medical visual grounding by introducing the first benchmark for sequential medical image grounding, covering eight VQA-style tasks across two paradigms and integrating data from 76 datasets and 10 modalities. It demonstrates that current MLLMs struggle with fine-grained grounding in image sequences, motivating the creation of MedSG-188K for instruction tuning and MedSeq-Grounder to advance research in this area. The work provides a comprehensive evaluation framework, a large-scale instructional dataset, and a purpose-built model, aiming to spur progress in reliable, cross-image clinical reasoning. Together, these contributions offer a practical foundation for improving sequential grounding in medical imaging and may enhance clinical decision support and interpretability of multimodal AI systems.

Abstract

Visual grounding is essential for precise perception and reasoning in multimodal large language models (MLLMs), especially in medical imaging domains. While existing medical visual grounding benchmarks primarily focus on single-image scenarios, real-world clinical applications often involve sequential images, where accurate lesion localization across different modalities and temporal tracking of disease progression (e.g., pre- vs. post-treatment comparison) require fine-grained cross-image semantic alignment and context-aware reasoning. To remedy the underrepresentation of image sequences in existing medical visual grounding benchmarks, we propose MedSG-Bench, the first benchmark tailored for Medical Image Sequences Grounding. It comprises eight VQA-style tasks, formulated into two paradigms of the grounding tasks, including 1) Image Difference Grounding, which focuses on detecting change regions across images, and 2) Image Consistency Grounding, which emphasizes detection of consistent or shared semantics across sequential images. MedSG-Bench covers 76 public datasets, 10 medical imaging modalities, and a wide spectrum of anatomical structures and diseases, totaling 9,630 question-answer pairs. We benchmark both general-purpose MLLMs (e.g., Qwen2.5-VL) and medical-domain specialized MLLMs (e.g., HuatuoGPT-vision), observing that even the advanced models exhibit substantial limitations in medical sequential grounding tasks. To advance this field, we construct MedSG-188K, a large-scale instruction-tuning dataset tailored for sequential visual grounding, and further develop MedSeq-Grounder, an MLLM designed to facilitate future research on fine-grained understanding across medical sequential images. The benchmark, dataset, and model are available at https://huggingface.co/MedSG-Bench
Paper Structure (36 sections, 2 equations, 6 figures, 3 tables)

This paper contains 36 sections, 2 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Multimodal Large Language Models
  4. Medical MLLM Benchmarks
  5. MedSG-Bench
  6. Data Collection and Preprocessing
  7. Dataset Review and Selection
  8. Standardization
  9. VQA tasks definition and generation
  10. Image Difference Grounding
  11. Task 1: Registered Difference Grounding
  12. Task 2: Non-registered Difference Grounding
  13. Image Consistency Grounding
  14. Task 3: Multi-View Grounding
  15. Task 4: Object Tracking
  16. ...and 21 more sections

Figures (6)

  • Figure 1: Examples of medical image sequences grounding.
  • Figure 2: Detailed statistics of MedSG-Bench.
  • Figure 3: An illustration of medical image sequences grounding tasks included in MedSG-Bench.
  • Figure 4: Overview of the MedSG-Bench construction protocol.
  • Figure 5: Proportions of image sequence length (left), data distribution across tasks (middle), and target-to-image size ratios (right) in MedSG-Bench.
  • ...and 1 more figures