Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fast and Efficient Modern BERT based Text-Conditioned Diffusion Model for Medical Image Segmentation

Venkata Siddharth Dhara, Pawan Kumar

TL;DR

Medical image segmentation often requires dense pixel-level annotations, which hinder clinical deployment. This work introduces FastTextDiff, a text-conditioned diffusion framework that uses ModernBERT to fuse clinical text with image features via cross-modal attention, enabling label-efficient segmentation. By pretraining ModernBERT on MIMIC-III/IV and integrating it into a diffusion backbone, FastTextDiff achieves competitive or state-of-the-art Dice/IoU on MoNuSeg, QaTa-COV19, and MosMedDataPlus while significantly reducing training and inference time compared with baselines such as TextDiff and larger vision-language models. The results support the practical potential of multimodal medical imaging pipelines and highlight the value of clinical text guidance for scalable segmentation in data-scarce clinical settings.

Abstract

In recent times, denoising diffusion probabilistic models (DPMs) have proven effective for medical image generation and denoising, and as representation learners for downstream segmentation. However, segmentation performance is limited by the need for dense pixel-wise labels, which are expensive, time-consuming, and require expert knowledge. We propose FastTextDiff, a label-efficient diffusion-based segmentation model that integrates medical text annotations to enhance semantic representations. Our approach uses ModernBERT, a transformer capable of processing long clinical notes, to tightly link textual annotations with semantic content in medical images. Trained on MIMIC-III and MIMIC-IV, ModernBERT encodes clinical knowledge that guides cross-modal attention between visual and textual features. This study validates ModernBERT as a fast, scalable alternative to Clinical BioBERT in diffusion-based segmentation pipelines and highlights the promise of multi-modal techniques for medical image analysis. By replacing Clinical BioBERT with ModernBERT, FastTextDiff benefits from FlashAttention 2, an alternating attention mechanism, and a 2-trillion-token corpus, improving both segmentation accuracy and training efficiency over traditional diffusion-based models.

A Fast and Efficient Modern BERT based Text-Conditioned Diffusion Model for Medical Image Segmentation

TL;DR

Medical image segmentation often requires dense pixel-level annotations, which hinder clinical deployment. This work introduces FastTextDiff, a text-conditioned diffusion framework that uses ModernBERT to fuse clinical text with image features via cross-modal attention, enabling label-efficient segmentation. By pretraining ModernBERT on MIMIC-III/IV and integrating it into a diffusion backbone, FastTextDiff achieves competitive or state-of-the-art Dice/IoU on MoNuSeg, QaTa-COV19, and MosMedDataPlus while significantly reducing training and inference time compared with baselines such as TextDiff and larger vision-language models. The results support the practical potential of multimodal medical imaging pipelines and highlight the value of clinical text guidance for scalable segmentation in data-scarce clinical settings.

Abstract

In recent times, denoising diffusion probabilistic models (DPMs) have proven effective for medical image generation and denoising, and as representation learners for downstream segmentation. However, segmentation performance is limited by the need for dense pixel-wise labels, which are expensive, time-consuming, and require expert knowledge. We propose FastTextDiff, a label-efficient diffusion-based segmentation model that integrates medical text annotations to enhance semantic representations. Our approach uses ModernBERT, a transformer capable of processing long clinical notes, to tightly link textual annotations with semantic content in medical images. Trained on MIMIC-III and MIMIC-IV, ModernBERT encodes clinical knowledge that guides cross-modal attention between visual and textual features. This study validates ModernBERT as a fast, scalable alternative to Clinical BioBERT in diffusion-based segmentation pipelines and highlights the promise of multi-modal techniques for medical image analysis. By replacing Clinical BioBERT with ModernBERT, FastTextDiff benefits from FlashAttention 2, an alternating attention mechanism, and a 2-trillion-token corpus, improving both segmentation accuracy and training efficiency over traditional diffusion-based models.

Paper Structure

This paper contains 22 sections, 5 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology: FastTextDiff
  4. Background: Diffusion Models
  5. Forward Process
  6. Backward Process
  7. Denoiser Network
  8. Crossmodal Attention
  9. Mechanism
  10. FastTextDiff Framework
  11. Architecture Overview
  12. Loss Function
  13. Inputs and Outputs
  14. Numerical Experiments
  15. Experimental Setup
  16. ...and 7 more sections

Figures (3)

  • Figure 1: Architecture of our proposed FastTextDiff method, integrating ModernBERT text embeddings with a U-Net based diffusion model via cross-modal attention for text-conditioned segmentation.
  • Figure 2: Loss Comparison across 2 datasets for TextDiff (baseline BERT) and FastTextDiff (ModernBERT). FastTextDiff generally shows faster convergence or reaches a lower loss plateau.
  • Figure 3: Visual segmentation comparisons on representative images from MoNuSeg kumar2019multi, QaTa-COV19 degerli2022osegnet, and MosMedData+ li2021dual. FastTextDiff often produces segmentations similar to the ground truth compared to other state-of-the-art methods.