MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine

TL;DR

MedTrinity-25M introduces a scalable, multimodal medical dataset that pairs images with automated multigranular ROI descriptions without requiring paired text. The authors combine ROI grounding, retrieval-augmented generation, and a specialized MLLM (LLaVA-Tri) to produce rich image-ROI-description triplets across 10 modalities and 65+ diseases. They demonstrate strong VQA performance gains on VQA-RAD, SLAKE, and PathVQA when pretraining with MedTrinity-25M, and show broad improvements across models, underscoring the dataset’s potential as a medical foundation for multimodal AI. The work enables scalable pretraining and downstream medical vision-language tasks, with the dataset publicly available for research.

Abstract

This paper introduces MedTrinity-25M, a comprehensive, large-scale multimodal dataset for medicine, covering over 25 million images across 10 modalities with multigranular annotations for more than 65 diseases. These multigranular annotations encompass both global information, such as modality and organ detection, and local information like ROI analysis, lesion texture, and region-wise correlations. Unlike the existing multimodal datasets, which are limited by the availability of image-text pairs, we have developed the first automated pipeline that scales up multimodal data by generating multigranular visual and textual annotations in the form of image-ROI-description triplets without the need for any paired text descriptions. Specifically, data from over 30 different sources have been collected, preprocessed, and grounded using domain-specific expert models to identify ROIs related to abnormal regions. We then build a comprehensive knowledge base and prompt multimodal large language models to perform retrieval-augmented generation with the identified ROIs as guidance, resulting in multigranular textual descriptions. Compared to existing datasets, MedTrinity-25M provides the most enriched annotations, supporting a comprehensive range of multimodal tasks such as captioning and report generation, as well as vision-centric tasks like classification and segmentation. We propose LLaVA-Tri by pretraining LLaVA on MedTrinity-25M, achieving state-of-the-art performance on VQA-RAD, SLAKE, and PathVQA, surpassing representative SOTA multimodal large language models. Furthermore, MedTrinity-25M can also be utilized to support large-scale pre-training of multimodal medical AI models, contributing to the development of future foundation models in the medical domain. We will make our dataset available.

Figures (15)

• Figure 1: Qualitative comparison with different types of dataset.
• Figure 1: Comparison of types of annotations in MedTrinity-25M with other multimodal datasets.
• Figure 2: Data construction pipeline.1) Data processing, including metadata integration to generate coarse caption, ROI locating, and medical knowledge collection. 2) Multigranular Textual Description Generation based on processed data.
• Figure 3: A qualitative comparison example of generated textual description with and without coarse caption. Without a coarse caption, MLLMs fails to detect diseases. On the contrary, providing a caption mentioning "COVID-19" allows MLLMs to identify and categorize the disease, facilitating further analysis.
• Figure 4: A qualitative comparison example of generated textual description with and without locating ROIs. Without ROIs, the caption offers only a brief global analysis; with ROIs, MLLMs conducts detailed local analysis and assesses the impact of lesion ROIs on adjacent normal regions.