Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Tasks

TL;DR

This work tackles the lack of region-level grounding in medical MLLMs by introducing MedRegA, a bilingual generalist model capable of image- and region-level vision-language tasks across eight modalities. It builds MedRegInstruct, a large region-centric dataset consisting of Region-Text and Grounded-Report pairs, and combines it with diverse medical multimodal corpora to train MedRegA. The model employs two-stage training and Regional CoT to enhance spatial grounding and interpretability, achieving superior performance in VQA, report generation, and classification, as well as robust region identification and grounded reporting across modalities and languages. The results demonstrate improved interpretability and clinician-facing interactivity, with strong region-to-text and text-to-region alignment, making the system suitable for diverse biomedical tasks and bilingual clinical documentation.

Abstract

Several medical Multimodal Large Languange Models (MLLMs) have been developed to address tasks involving visual images with textual instructions across various medical modalities, achieving impressive results. Most current medical generalist models are region-agnostic, treating the entire image as a holistic representation. However, they struggle to identify which specific regions they are focusing on when generating a sentence. To mimic the behavior of doctors, who typically begin by reviewing the entire image before concentrating on specific regions for a thorough evaluation, we aim to enhance the capability of medical MLLMs in understanding anatomical regions within entire medical scans. To achieve it, we first formulate Region-Centric tasks and construct a large-scale dataset, MedRegInstruct, to incorporate regional information into training. Combining our collected dataset with other medical multimodal corpora for training, we propose a Region-Aware medical MLLM, MedRegA, which is the first bilingual generalist medical AI system to simultaneously handle image-level and region-level medical vision-language tasks across a broad range of modalities. Our MedRegA not only enables three region-centric tasks, but also achieves the best performance for visual question answering, report generation and medical image classification over 8 modalities, showcasing significant versatility. Experiments demonstrate that our model can not only accomplish powerful performance across various medical vision-language tasks in bilingual settings, but also recognize and detect structures in multimodal medical scans, boosting the interpretability and user interactivity of medical MLLMs. Our project page is https://medrega.github.io.

Figures (19)

• Figure 1: MedRegA, an interpretable bilingual generalist model for diverse biomedical tasks, represented by its outstanding ability to leverage regional information. MedRegA can perceive 8 modalities covering almost all the body parts, showcasing significant versatility.
• Figure 2: The significance of Region-Centric ability. (a) Comparison between the region-agnostic model (MedDr) and the region-centric MedRegA in analyzing lesion area within the medical scan. (b) Performance comparison of prompting the model with and without regional information on five benchmarks of Visual Question Answering (VQA) and classification tasks.
• Figure 3: Data Construction Pipeline for Report-Grounded Dataset. The automatic data processing procedure is composed of three steps: Image-Report Pair Construction, Report Refinement, and Structure Detection.
• Figure 4: An illustrative example of performing region-centric tasks with MedRegA.
• Figure 5: Comparison of traditional inference and generation pipeline with Regional CoT.