Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

Aryan Das, Tanishq Rachamalla, Koushik Biswas, Swalpa Kumar Roy, Vinay Kumar Verma

TL;DR

This work tackles uncertainty in multimodal vision-language segmentation for medical imaging by fusing radiological images with accompanying clinical text. It introduces a lightweight Modality Decoding Attention Block (MoDAB) and a State Space Mixer (SSMix) for efficient cross-modal fusion, guided by the Spectral-Entropic Uncertainty (SEU) Loss that unifies spatial, spectral, and uncertainty considerations. The architecture combines a visual ConvNeXt encoder with a frozen text encoder (BioViL CXR-BERT) and a four-stage decoder to produce precise segmentation. Empirical results across QaTa-COV19, MosMed++, and Kvasir-SEG demonstrate state-of-the-art accuracy with favorable computational efficiency, underscoring the value of incorporating uncertainty modeling and structured modality alignment in medical VLS tasks. The work provides code for reproducibility and potential clinical deployment.

Abstract

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS

Uncertainty-Aware Vision-Language Segmentation for Medical Imaging

TL;DR

This work tackles uncertainty in multimodal vision-language segmentation for medical imaging by fusing radiological images with accompanying clinical text. It introduces a lightweight Modality Decoding Attention Block (MoDAB) and a State Space Mixer (SSMix) for efficient cross-modal fusion, guided by the Spectral-Entropic Uncertainty (SEU) Loss that unifies spatial, spectral, and uncertainty considerations. The architecture combines a visual ConvNeXt encoder with a frozen text encoder (BioViL CXR-BERT) and a four-stage decoder to produce precise segmentation. Empirical results across QaTa-COV19, MosMed++, and Kvasir-SEG demonstrate state-of-the-art accuracy with favorable computational efficiency, underscoring the value of incorporating uncertainty modeling and structured modality alignment in medical VLS tasks. The work provides code for reproducibility and potential clinical deployment.

Abstract

We introduce a novel uncertainty-aware multimodal segmentation framework that leverages both radiological images and associated clinical text for precise medical diagnosis. We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion and long-range dependency modelling. To guide learning under ambiguity, we propose the Spectral-Entropic Uncertainty (SEU) Loss, which jointly captures spatial overlap, spectral consistency, and predictive uncertainty in a unified objective. In complex clinical circumstances with poor image quality, this formulation improves model reliability. Extensive experiments on various publicly available medical datasets, QATA-COVID19, MosMed++, and Kvasir-SEG, demonstrate that our method achieves superior segmentation performance while being significantly more computationally efficient than existing State-of-the-Art (SoTA) approaches. Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks. Code: https://github.com/arya-domain/UA-VLS
Paper Structure (17 sections, 28 equations, 3 figures, 2 tables)

This paper contains 17 sections, 28 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Modalities Encoding
  5. Modality Decoding Attention Block (MoDAB)
  6. State Space Mixer (SSMix)
  7. Decoder
  8. Objective Function
  9. Experiments
  10. Experimental Setup
  11. Datasets
  12. Training Details
  13. Results and Discussion
  14. Qualitative and Quantitative Analysis
  15. Computational Efficiency Analysis
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of the proposed architecture. The model integrates visual and frozen text encoders and the Modality Decoding Attention Block (MoDAB), which incorporates Self-Attention and Cross-Attention along with a State Space Mixer (SSMix) for efficient multimodal fusion. The decoder reconstructs segmentation masks from the fused features through a multi-stage upsampling pathway.
  • Figure 2: Comparison of Grad-CAM-Based Attention Visualizations Between the Proposed Model and Baseline methods
  • Figure 3: Qualitative Comparison of Predicted Segmentation Maps with Baseline Models