MIMO: A medical vision language model with visual referring multimodal input and pixel grounding multimodal output

TL;DR

This work tackles two limitations of medical vision-language models: reliance on text-only inputs and text-only outputs that miss region-level grounding. It introduces MIMO, a unified MVLM that accepts visual prompts and generates outputs that are linked to pixel-level segmentations, facilitated by a visual prompting input aligner and a segmentation-grounded decoder. To train and evaluate the model, the authors create MIMOSeg, a large-scale medical multimodal dataset with 895K samples spanning diverse modalities and four reasoning/QA perspectives. Experiments demonstrate that MIMO uniquely combines visual referring and pixel grounding capabilities, outperforming prior models on medical VQA and segmentation-grounding tasks, with potential impact on clinically grounded AI tools.

Abstract

Currently, medical vision language models are widely used in medical vision question answering tasks. However, existing models are confronted with two issues: for input, the model only relies on text instructions and lacks direct understanding of visual clues in the image; for output, the model only gives text answers and lacks connection with key areas in the image. To address these issues, we propose a unified medical vision language model MIMO, with visual referring Multimodal Input and pixel grounding Multimodal Output. MIMO can not only combine visual clues and textual instructions to understand complex medical images and semantics, but can also ground medical terminologies in textual output within the image. To overcome the scarcity of relevant data in the medical field, we propose MIMOSeg, a comprehensive medical multimodal dataset including 895K samples. MIMOSeg is constructed from four different perspectives, covering basic instruction following and complex question answering with multimodal input and multimodal output. We conduct experiments on several downstream medical multimodal tasks. Extensive experimental results verify that MIMO can uniquely combine visual referring and pixel grounding capabilities, which are not available in previous models.

Figures (14)

• Figure 1: An illustration of visual referring and pixel grounding capabilities.
• Figure 2: The overall architecture of MIMO. The model consists of an image encoder, a visual prompt encoder, a LLM, a segmentation vision encoder and a mask decoder. While accepting optional visual input, the model can also provide visual segmentation results associated with the medical entities in the text response.
• Figure 3: The construction pipeline of MIMOSeg. The pipeline inclues data collection, knowledge retrieval, prompt construction and QA generation. The bottom of the figure shows example data from four perspectives of MIMOSeg.
• Figure 4: Qualitative analysis of experimental results from three perspectives. We compare the results with LLaVA-Med and HuatuoGPT-Vision. MIMO can generate segmentation masks with relevant medical entities while outputting the answer.
• Figure 5: Held-in and held-out experimental results with different ratios of training data.