MedMO: Grounding and Understanding Multimodal Large Language Model for Medical Images

TL;DR

MedMO tackles the gap between general multimodal LLMs and domain-specific medical understanding by building an open-source medical vision-language model with strong grounding. It deploys a four-stage post-training recipe—general SFT, high-resolution grounding, instruction tuning, and reinforcement learning with a verifiable bounding-box reward—to align vision encoders with a medical language backbone and enforce spatial reasoning. Across VQA, QA, report generation, and grounding benchmarks, MedMO shows state-of-the-art or near-SOTA performance and demonstrates robust cross-domain generalization across radiology, ophthalmology, and pathology. The work provides a scalable, transparent roadmap for developing reliable open medical VLMs and releases 4B and 8B versions for broader impact.

Abstract

Multimodal large language models (MLLMs) have rapidly advanced, yet their adoption in medicine remains limited by gaps in domain coverage, modality alignment, and grounded reasoning. In this work, we introduce MedMO, a medical foundation model built upon a generalized MLLM architecture and trained exclusively on large-scale, domain-specific data. MedMO follows a multi-stage training recipe: (i) cross-modal pretraining to align heterogeneous visual encoders with a medical language backbone; (ii) instruction tuning on multi-task supervision that spans captioning, VQA, report generation, retrieval, and grounded disease localization with bounding boxes; and (iii) reinforcement learning with verifiable rewards that combine factuality checks with a box-level GIoU reward to strengthen spatial grounding and step-by-step reasoning in complex clinical scenarios. MedMO consistently outperforms strong open-source medical MLLMs across multiple modalities and tasks. On VQA benchmarks, MedMO achieves an average accuracy improvement of +13.7% over the baseline and performs within 1.9% of the SOTA Fleming-VL. For text-based QA, it attains +6.9% over the baseline and +14.5% over Fleming-VL. In medical report generation, MedMO delivers significant gains in both semantic and clinical accuracy. Moreover, it exhibits strong grounding capability, achieving an IoU improvement of +40.4 over the baseline and +37.0% over Fleming-VL, underscoring its robust spatial reasoning and localization performance. Evaluations across radiology, ophthalmology, and pathology-microscopy confirm MedMO's broad cross-modality generalization. We release two versions of MedMO: 4B and 8B. Project is available at https://genmilab.github.io/MedMO-Page

Figures (20)

• Figure 1: Benchmark performance of MedMO-8B across diverse medical imaging tasks, including VQA, QA, report generation, and grounding. MedMO achieves consistent gains over prior models, with improvements of +1.3% on MMMU-Med, +0.1% on MedXQA, +0.7% on MMLU-Med, +24.3% on MedQA, +5.1% on MIMIC-CXR, and a substantial +43.8 IoU on Bacteria segmentation. The large boost in Bacteria IoU stems from the incorporation of fine-grained grounding supervision and high-resolution microscopy data, highlighting MedMO’s enhanced spatial reasoning and localization capabilities.
• Figure 2: Overview of the multi-stage training pipeline for medical image analysis. The workflow consists of three main capabilities: (Top row) VQA/QA for identifying abnormalities in medical images, Grounding for spatial localization of detected features with bounding box coordinates, and Report generation for producing detailed medical reports. (Bottom) The training pipeline progresses through four sequential stages: (1) Large-scale training on 18.5M image-text pairs at 768$\times$768 resolution for global image-text alignment, (2) High-resolution training on 3M samples at 1280$\times$1280 resolution to enhance spatial localization and fine-grained visual grounding, (3) Instruction tuning on 4.3M samples covering captioning, diagnosis, and report summarization tasks to align responses with human-style medical instruction following, and (4) Medical-oriented reinforcement learning on 300K samples optimized using four reward signals: label accuracy, bounding box IoU, tag count, and soft overlap punishment. The complete pipeline for the MedMO-8B.
• Figure 3: Qualitative comparison across diverse medical and visual question-answering tasks. Each block shows the ground truth, model predictions from Fleming-VL-8B (current Medical SOTA), Qwen3-VL (Baseline), and MedMO, and highlights textual or spatial alignment. MedMO provides more accurate medical understanding and localization in both diagnostic accuracy and clinical reasoning.
• Figure 4: Composition of the unified multi-modal medical dataset comprising diverse imaging modalities and biological systems.
• Figure 5: Performance across post-training stages on radiology datasets. MedMO exhibits consistent gains in diagnostic accuracy and localization across IU-Xray, MIMIC-CXR, CheXpert, and MedTrinity datasets. The sharp improvement at Stage 2 highlights the benefit of alignment tuning with medical reasoning objectives.