Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology

Chengrun Li, Corentin Royer, Haozhe Luo, Bastian Wittmann, Xia Li, Ibrahim Hamamci, Sezgin Er, Anjany Sekuboyina, Bjoern Menze

TL;DR

RadDiagSeg-M tackles the lack of joint text-and-mask outputs in radiology VLMs by introducing RadDiagSeg-D, a 28k-sample three-step VQA-and-segmentation dataset, and RadDiagSeg-M, a three-component vision-language model that jointly detects abnormalities, provides diagnosis, and outputs multiple segmentation masks. The model leverages a MedSAM-based vision backbone, a BiomedCLIP-informed multimodal LM with LoRA fine-tuning, and a MedSAM mask decoder, trained end-to-end with a combined loss that fuses language and segmentation objectives. Empirical results show state-of-the-art performance on radiology VQA tasks and the first capable solution for the full VQA-Seg task, while establishing a robust baseline for the RadDiagSeg-D benchmark. The work also provides a benchmarking tool to standardize evaluation of multi-target text-and-mask generation in radiology, supporting more interpretable and actionable clinical AI assistance.

Abstract

Most current medical vision language models struggle to jointly generate diagnostic text and pixel-level segmentation masks in response to complex visual questions. This represents a major limitation towards clinical application, as assistive systems that fail to provide both modalities simultaneously offer limited value to medical practitioners. To alleviate this limitation, we first introduce RadDiagSeg-D, a dataset combining abnormality detection, diagnosis, and multi-target segmentation into a unified and hierarchical task. RadDiagSeg-D covers multiple imaging modalities and is precisely designed to support the development of models that produce descriptive text and corresponding segmentation masks in tandem. Subsequently, we leverage the dataset to propose a novel vision-language model, RadDiagSeg-M, capable of joint abnormality detection, diagnosis, and flexible segmentation. RadDiagSeg-M provides highly informative and clinically useful outputs, effectively addressing the need to enrich contextual information for assistive diagnosis. Finally, we benchmark RadDiagSeg-M and showcase its strong performance across all components involved in the task of multi-target text-and-mask generation, establishing a robust and competitive baseline.

RadDiagSeg-M: A Vision Language Model for Joint Diagnosis and Multi-Target Segmentation in Radiology

TL;DR

RadDiagSeg-M tackles the lack of joint text-and-mask outputs in radiology VLMs by introducing RadDiagSeg-D, a 28k-sample three-step VQA-and-segmentation dataset, and RadDiagSeg-M, a three-component vision-language model that jointly detects abnormalities, provides diagnosis, and outputs multiple segmentation masks. The model leverages a MedSAM-based vision backbone, a BiomedCLIP-informed multimodal LM with LoRA fine-tuning, and a MedSAM mask decoder, trained end-to-end with a combined loss that fuses language and segmentation objectives. Empirical results show state-of-the-art performance on radiology VQA tasks and the first capable solution for the full VQA-Seg task, while establishing a robust baseline for the RadDiagSeg-D benchmark. The work also provides a benchmarking tool to standardize evaluation of multi-target text-and-mask generation in radiology, supporting more interpretable and actionable clinical AI assistance.

Abstract

Most current medical vision language models struggle to jointly generate diagnostic text and pixel-level segmentation masks in response to complex visual questions. This represents a major limitation towards clinical application, as assistive systems that fail to provide both modalities simultaneously offer limited value to medical practitioners. To alleviate this limitation, we first introduce RadDiagSeg-D, a dataset combining abnormality detection, diagnosis, and multi-target segmentation into a unified and hierarchical task. RadDiagSeg-D covers multiple imaging modalities and is precisely designed to support the development of models that produce descriptive text and corresponding segmentation masks in tandem. Subsequently, we leverage the dataset to propose a novel vision-language model, RadDiagSeg-M, capable of joint abnormality detection, diagnosis, and flexible segmentation. RadDiagSeg-M provides highly informative and clinically useful outputs, effectively addressing the need to enrich contextual information for assistive diagnosis. Finally, we benchmark RadDiagSeg-M and showcase its strong performance across all components involved in the task of multi-target text-and-mask generation, establishing a robust and competitive baseline.
Paper Structure (47 sections, 9 equations, 10 figures, 7 tables)

This paper contains 47 sections, 9 equations, 10 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Methods
  3. Segmentation Models in Medical Images
  4. Medical Vision Language Models
  5. Medical VLMs with Segmentation Mask Outputs
  6. Methods
  7. RadDiagSeg-M: Model Architecture
  8. Vision backbone.
  9. Multimodal LM.
  10. Mask decoder.
  11. Training objectives.
  12. RadDiagSeg-M: Training Data and Tasks
  13. Referring segmentation task (Ref-Seg).
  14. Visual question answering task (VQA).
  15. VQA segmentation task (VQA-Seg).
  16. ...and 32 more sections

Figures (10)

  • Figure 1: Overview. RadDiagSeg-M is capable of jointly detecting and diagnosing abnormality, and providing multi-target segmentation masks.
  • Figure 2: Architecture of RadDiagSeg-M. Our model supports multi-target flexible segmentation.
  • Figure 3: Three types of tasks in training. Notably, the VQA Segmentation (VQA-Seg) is a novel complex task that involves three steps: textual tasks of detection and diagnosis, and a multi-target segmentation task.
  • Figure 4: Qualitative results for the VQA-Seg and VQA tasks. RadDiagSeg-M is the only model answering all three questions correctly while providing multiple well-referred masks following the instructions. The results underpin the capability of RadDiagSeg-M at multi-target text-and-mask generation.
  • Figure 5: Box plot comparing performances of models on the Ref-Seg task across three modalities. RadDiagSeg-M significantly outperforms the baseline methods.
  • ...and 5 more figures