Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedVLSynther: Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs

Xiaoke Huang, Ningsen Wang, Hui Liu, Xianfeng Tang, Yuyin Zhou

TL;DR

MedVLSynther tackles the training-data bottleneck in medical VQA by introducing a rubric-guided generator–verifier framework that synthesizes high-quality, auditable MC-VQA items from open biomedical literature. The resulting MedSynVQA dataset (13,087 questions over 14,803 images across 13 modalities and 28 anatomical regions) enables open-weight LMMs to achieve state-of-the-art averages on six medical VQA benchmarks via reinforcement learning with verifiable rewards. The pipeline employs a three-stage verifier (essential gates, bonus criteria, penalties) to ensure self-containment, image-text grounding, and clinically valid reasoning, yielding high-precision data with no leakage into evaluation sets. By operating entirely on open data and models, MedVLSynther provides a reproducible, privacy-preserving path to scalable medical VQA supervision that can accelerate multimodal biomedical intelligence.

Abstract

Large Multimodal Models (LMMs) are increasingly capable of answering medical questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guided generator-verifier framework that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive options under a machine-checkable JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image-text consistency), awards fine-grained positive points, and penalizes common failure modes before acceptance. Applying this pipeline to PubMed Central yields MedSynVQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with reinforcement learning using verifiable rewards improves accuracy across six medical VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up to 77.57 on VQA-RAD and 67.76 on PathVQA, outperforming strong medical LMMs. A Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers an auditable, reproducible, and privacy-preserving path to scalable medical VQA training data.

MedVLSynther: Synthesizing High-Quality Visual Question Answering from Medical Documents with Generator-Verifier LMMs

TL;DR

MedVLSynther tackles the training-data bottleneck in medical VQA by introducing a rubric-guided generator–verifier framework that synthesizes high-quality, auditable MC-VQA items from open biomedical literature. The resulting MedSynVQA dataset (13,087 questions over 14,803 images across 13 modalities and 28 anatomical regions) enables open-weight LMMs to achieve state-of-the-art averages on six medical VQA benchmarks via reinforcement learning with verifiable rewards. The pipeline employs a three-stage verifier (essential gates, bonus criteria, penalties) to ensure self-containment, image-text grounding, and clinically valid reasoning, yielding high-precision data with no leakage into evaluation sets. By operating entirely on open data and models, MedVLSynther provides a reproducible, privacy-preserving path to scalable medical VQA supervision that can accelerate multimodal biomedical intelligence.

Abstract

Large Multimodal Models (LMMs) are increasingly capable of answering medical questions that require joint reasoning over images and text, yet training general medical VQA systems is impeded by the lack of large, openly usable, high-quality corpora. We present MedVLSynther, a rubric-guided generator-verifier framework that synthesizes high-quality multiple-choice VQA items directly from open biomedical literature by conditioning on figures, captions, and in-text references. The generator produces self-contained stems and parallel, mutually exclusive options under a machine-checkable JSON schema; a multi-stage verifier enforces essential gates (self-containment, single correct answer, clinical validity, image-text consistency), awards fine-grained positive points, and penalizes common failure modes before acceptance. Applying this pipeline to PubMed Central yields MedSynVQA: 13,087 audited questions over 14,803 images spanning 13 imaging modalities and 28 anatomical regions. Training open-weight LMMs with reinforcement learning using verifiable rewards improves accuracy across six medical VQA benchmarks, achieving averages of 55.85 (3B) and 58.15 (7B), with up to 77.57 on VQA-RAD and 67.76 on PathVQA, outperforming strong medical LMMs. A Ablations verify that both generation and verification are necessary and that more verified data consistently helps, and a targeted contamination analysis detects no leakage from evaluation suites. By operating entirely on open literature and open-weight models, MedVLSynther offers an auditable, reproducible, and privacy-preserving path to scalable medical VQA training data.

Paper Structure

This paper contains 31 sections, 2 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodal medical VQA.
  4. Synthetic data generation for multimodal medical VQA.
  5. Multimodal models, medical adaptation, and reasoning.
  6. MedVLSynther and MedSynVQA
  7. Data source, extraction, and pre‑filtering
  8. Source.
  9. Pre‑filtering.
  10. Choice of generator and verifier LMMs.
  11. Rubric‑based, context‑aware VQA generation
  12. Multi‑stage, rubric‑based, context‑aware verification
  13. Stage‑1: Essential screening (hard gate).
  14. Stage‑2: Fine‑grained positive criteria (bonus points).
  15. Stage‑3: Penalty criteria (error hunting).
  16. ...and 16 more sections

Figures (6)

  • Figure 1: (a) Stage‑1 generation: a rubric‑guided LMM converts PubMed figures and captions into multiple‑choice VQA items. (b) Stage‑2 verification: a multi‑stage, rubric‑based LMM verifier screens items and filters low‑quality ones. (c) Training open‑weight students (3B/7B) on MedSynVQA yields consistent gains over strong medical LMM baselines.
  • Figure 2: MedSynVQA statistics: 1) Dataset distributions for question type, imaging modality, and anatomy. 2) Word cloud for generated questions.
  • Figure 3: From PubMed documents we extract figures and reference text, then apply (a) extraction and pre‑filtering by primary/secondary tags; (b) rubric‑based, context‑aware generation with format constraints and question archetypes; (c) multi‑stage verification with essential, fine‑grained, and penalty criteria. Items are retained if their rubric score exceeds a threshold.
  • Figure 7: Examples of context-aware generation and leakage rejection by the verifier.
  • Figure :
  • ...and 1 more figures