MedGEN-Bench: Contextually entangled benchmark for open-ended multimodal medical generation

TL;DR

MedGEN-Bench addresses critical gaps in medical visual benchmarks by offering an open-ended, contextually entangled multimodal benchmark that jointly evaluates diagnostic text generation and clinically accurate image synthesis. It assembles 6,422 image-text pairs across six modalities and 16 tasks, organized into Visual Question Answering, Image Editing, and Contextual Multimodal Generation formats to force deep cross-modal reasoning. A novel three-tier evaluation framework combines pixel-level metrics, semantic text analysis, and expert-guided clinical relevance scoring, enabling robust assessment beyond surface metrics. Across extensive experiments with ten compositional frameworks, three unified models, and five LVLMs, compositional approaches excel in holistic generation while unified models exhibit cross-modal disconnections; contextual augmentation significantly improves instruction relevance. Overall, MedGEN-Bench establishes a principled, clinically grounded platform for advancing open-ended multimodal generation in medicine and guiding the development of clinically interpretable AI systems.

Abstract

As Vision-Language Models (VLMs) increasingly gain traction in medical applications, clinicians are progressively expecting AI systems not only to generate textual diagnoses but also to produce corresponding medical images that integrate seamlessly into authentic clinical workflows. Despite the growing interest, existing medical visual benchmarks present notable limitations. They often rely on ambiguous queries that lack sufficient relevance to image content, oversimplify complex diagnostic reasoning into closed-ended shortcuts, and adopt a text-centric evaluation paradigm that overlooks the importance of image generation capabilities. To address these challenges, we introduce MedGEN-Bench, a comprehensive multimodal benchmark designed to advance medical AI research. MedGEN-Bench comprises 6,422 expert-validated image-text pairs spanning six imaging modalities, 16 clinical tasks, and 28 subtasks. It is structured into three distinct formats: Visual Question Answering, Image Editing, and Contextual Multimodal Generation. What sets MedGEN-Bench apart is its focus on contextually intertwined instructions that necessitate sophisticated cross-modal reasoning and open-ended generative outputs, moving beyond the constraints of multiple-choice formats. To evaluate the performance of existing systems, we employ a novel three-tier assessment framework that integrates pixel-level metrics, semantic text analysis, and expert-guided clinical relevance scoring. Using this framework, we systematically assess 10 compositional frameworks, 3 unified models, and 5 VLMs.

Figures (6)

• Figure 1: Existing medical visual benchmarks suffer from ambiguous queries that are insufficiently related to certain medical image.
• Figure 2: Our pilot study reveals critical limitations in existing medical visual benchmarks.
• Figure 3: Overview of our MedGEN-Bench.
• Figure 4: Overview of the MedGEN-Bench construction pipeline, structured into four sequential phases: (1) Preprocessing (blue) employs a two-stage filtering mechanism, initiating with metadata-based coarse filtering followed by GPT-4o semantic validation to select candidate medical images and associated metadata; (2) Image Pair Synthesis (yellow) integrates deterministic operations with generative image transformations, incorporating human review to maintain clinical fidelity; (3) Text Pair Synthesis (red) leverages Qwen3-VL for semantic information extraction to populate task-specific templates, augmented by GPT-4o for contextual query integration with visual content; and (4) Human Refinement (green) ensures output quality through automated Vision-Language Model review and expert validation.
• Figure 5: Left: An overview of MedGEN-Bench statics. Right: Distribution analysis of textual content length for instructions and answers.