Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis

Junzhi Ning, Wei Li, Cheng Tang, Jiashi Lin, Chenglong Ma, Chaoyang Zhang, Jiyao Liu, Ying Chen, Shujian Gao, Lihao Liu, Yuandong Pu, Huihui Xu, Chenhui Gou, Ziyan Huang, Yi Xin, Qi Qin, Zhongying Deng, Diping Song, Bin Fu, Guang Yang, Yuanfeng Ji, Tianbin Li, Yanzhou Su, Jin Ye, Shixiang Tang, Ming Hu, Junjun He

TL;DR

UniMedVL addresses fragmentation in medical AI by introducing the OKA paradigm (Observation-Knowledge-Analysis) to unify data, knowledge, and tasks within a single framework. It builds UniMed-5M, a 5.6M-sample multimodal medical dataset, applies a Progressive Curriculum Learning scheme, and deploys UniMedVL, a unified architecture that jointly handles medical image understanding and generation via dual encoders and a mixture-of-transformer-experts. Empirical results show strong performance on understanding benchmarks and competitive generation quality across eight modalities, with bidirectional knowledge sharing enhancing both capabilities. This work provides a scalable, end-to-end pathway toward integrated medical AI workflows that align with clinical diagnostic processes.

Abstract

Medical diagnostic applications require models that can process multimodal medical inputs (images, patient histories, lab results) and generate diverse outputs including both textual reports and visual content (annotations, segmentation masks, and images). Despite this need, existing medical AI systems disrupt this unified process: medical image understanding models interpret images but cannot generate visual outputs, while medical image generation models synthesize images but cannot provide textual explanations. This leads to gaps in data representation, feature integration, and task-level multimodal capabilities. To this end, we propose a multi-level framework that draws inspiration from diagnostic workflows through the Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation level, we construct UniMed-5M, a dataset comprising over 5.6M samples that reformat diverse unimodal data into multimodal pairs for foundational observation. At the knowledge level, we propose Progressive Curriculum Learning that systematically introduces medical multimodal knowledge. At the analysis level, we introduce UniMedVL, the first medical unified multimodal model for the simultaneous analysis of image understanding and generation tasks within a single architecture. UniMedVL achieves superior performance on five medical image understanding benchmarks, while matching specialized models in generation quality across eight medical imaging modalities. Crucially, our unified architecture enables bidirectional knowledge sharing: generation tasks enhance visual understanding features, demonstrating that integrating traditionally separate capabilities within a single medical framework unlocks improvements across diverse medical vision-language tasks. Code is available at https://github.com/uni-medical/UniMedVL.

UniMedVL: Unifying Medical Multimodal Understanding And Generation Through Observation-Knowledge-Analysis

TL;DR

UniMedVL addresses fragmentation in medical AI by introducing the OKA paradigm (Observation-Knowledge-Analysis) to unify data, knowledge, and tasks within a single framework. It builds UniMed-5M, a 5.6M-sample multimodal medical dataset, applies a Progressive Curriculum Learning scheme, and deploys UniMedVL, a unified architecture that jointly handles medical image understanding and generation via dual encoders and a mixture-of-transformer-experts. Empirical results show strong performance on understanding benchmarks and competitive generation quality across eight modalities, with bidirectional knowledge sharing enhancing both capabilities. This work provides a scalable, end-to-end pathway toward integrated medical AI workflows that align with clinical diagnostic processes.

Abstract

Medical diagnostic applications require models that can process multimodal medical inputs (images, patient histories, lab results) and generate diverse outputs including both textual reports and visual content (annotations, segmentation masks, and images). Despite this need, existing medical AI systems disrupt this unified process: medical image understanding models interpret images but cannot generate visual outputs, while medical image generation models synthesize images but cannot provide textual explanations. This leads to gaps in data representation, feature integration, and task-level multimodal capabilities. To this end, we propose a multi-level framework that draws inspiration from diagnostic workflows through the Observation-Knowledge-Analysis (OKA) paradigm. Specifically, at the observation level, we construct UniMed-5M, a dataset comprising over 5.6M samples that reformat diverse unimodal data into multimodal pairs for foundational observation. At the knowledge level, we propose Progressive Curriculum Learning that systematically introduces medical multimodal knowledge. At the analysis level, we introduce UniMedVL, the first medical unified multimodal model for the simultaneous analysis of image understanding and generation tasks within a single architecture. UniMedVL achieves superior performance on five medical image understanding benchmarks, while matching specialized models in generation quality across eight medical imaging modalities. Crucially, our unified architecture enables bidirectional knowledge sharing: generation tasks enhance visual understanding features, demonstrating that integrating traditionally separate capabilities within a single medical framework unlocks improvements across diverse medical vision-language tasks. Code is available at https://github.com/uni-medical/UniMedVL.
Paper Structure (40 sections, 3 equations, 18 figures, 11 tables)

This paper contains 40 sections, 3 equations, 18 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Medical Multimodal Large Language Models
  4. Unified Multimodal Understanding and Generation Models
  5. Methodology
  6. Observation Level: UniMed-5M Dataset Construction
  7. Knowledge Level: Progressive Curriculum Learning
  8. Analysis Level: UniMedVL Unified Architecture
  9. Experiments
  10. Benchmarks and Baselines
  11. Performance of UniMedVL
  12. Ablation Study
  13. Medical Visual Understanding Performance
  14. Medical Image Generation Performance
  15. Interleaved Multimodal Tasks Performance
  16. ...and 25 more sections

Figures (18)

  • Figure 1: Overview of UniMedVL unified framework. Capabilities across medical image understanding and generation tasks and performance comparisons.
  • Figure 2: Overview of the proposed Observation–Knowledge framework. Observation: Covers data sources and modality coverage, quality control pipeline, and interleaved image-text task construction for building training data across different model stages. Knowledge: Refers to the progressive curriculum training paradigm, consisting of three stages that gradually equip the model with generalized capabilities on interleaved image-text tasks.
  • Figure 3: Visual Comparison of Performance across different training stages and modalities.(Left:) Stage-wise understanding accuracy performance. (Right:) Generation quality evolution with gFID reduction and BiomedCLIP score enhancement through different training stages.
  • Figure 4: Comprehensive visualization of UniMedVL multimodal capabilities. Demonstration of diverse medical imaging tasks, including text-to-image generation, virtual staining, super resolution, counterfactual generation, and cross-modal synthesis.
  • Figure 5: Qualitative comparison of VAE reconstruction quality across diverse medical imaging modalities. Visual examples demonstrating reconstruction fidelity across eight medical imaging modalities (CFP, CT, CXR, Endoscopy, HIS, MRI, OCT, Ultrasound) using the pretrained FLUX VAE without domain-specific fine-tuning.
  • ...and 13 more figures