Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation

Amaan Izhar, Nurul Japar, Norisma Idris, Ting Dang

TL;DR

This work tackles MIR across diverse imaging modalities by introducing MicarVLMoE, a three-part architecture combining a multiscale vision encoder, a latent dual-branch attention encoder, and a latent MoE decoder with gated fusion. The method achieves state-of-the-art clinical accuracy and cross-modal alignment on COVCTR, MMR, PGROSS, and ROCO, supported by extensive ablations and hyperparameter analyses. It demonstrates improved interpretability through attention and MoE routing visualizations, and broadens MIR evaluation beyond chest X-rays to CT, retina, MRI, and pathology. The approach holds practical impact for reliable, scalable radiology report generation and lays groundwork for future knowledge-graph integration and large-scale multimodal training.

Abstract

Medical image reporting (MIR) aims to generate structured clinical descriptions from radiological images. Existing methods struggle with fine-grained feature extraction, multimodal alignment, and generalization across diverse imaging types, often relying on vanilla transformers and focusing primarily on chest X-rays. We propose MicarVLMoE, a vision-language mixture-of-experts model with gated cross-aligned fusion, designed to address these limitations. Our architecture includes: (i) a multiscale vision encoder (MSVE) for capturing anatomical details at varying resolutions, (ii) a multihead dual-branch latent attention (MDLA) module for vision-language alignment through latent bottleneck representations, and (iii) a modulated mixture-of-experts (MoE) decoder for adaptive expert specialization. We extend MIR to CT scans, retinal imaging, MRI scans, and gross pathology images, reporting state-of-the-art results on COVCTR, MMR, PGROSS, and ROCO datasets. Extensive experiments and ablations confirm improved clinical accuracy, cross-modal alignment, and model interpretability. Code is available at https://github.com/AI-14/micar-vl-moe.

MicarVLMoE: A Modern Gated Cross-Aligned Vision-Language Mixture of Experts Model for Medical Image Captioning and Report Generation

TL;DR

This work tackles MIR across diverse imaging modalities by introducing MicarVLMoE, a three-part architecture combining a multiscale vision encoder, a latent dual-branch attention encoder, and a latent MoE decoder with gated fusion. The method achieves state-of-the-art clinical accuracy and cross-modal alignment on COVCTR, MMR, PGROSS, and ROCO, supported by extensive ablations and hyperparameter analyses. It demonstrates improved interpretability through attention and MoE routing visualizations, and broadens MIR evaluation beyond chest X-rays to CT, retina, MRI, and pathology. The approach holds practical impact for reliable, scalable radiology report generation and lays groundwork for future knowledge-graph integration and large-scale multimodal training.

Abstract

Medical image reporting (MIR) aims to generate structured clinical descriptions from radiological images. Existing methods struggle with fine-grained feature extraction, multimodal alignment, and generalization across diverse imaging types, often relying on vanilla transformers and focusing primarily on chest X-rays. We propose MicarVLMoE, a vision-language mixture-of-experts model with gated cross-aligned fusion, designed to address these limitations. Our architecture includes: (i) a multiscale vision encoder (MSVE) for capturing anatomical details at varying resolutions, (ii) a multihead dual-branch latent attention (MDLA) module for vision-language alignment through latent bottleneck representations, and (iii) a modulated mixture-of-experts (MoE) decoder for adaptive expert specialization. We extend MIR to CT scans, retinal imaging, MRI scans, and gross pathology images, reporting state-of-the-art results on COVCTR, MMR, PGROSS, and ROCO datasets. Extensive experiments and ablations confirm improved clinical accuracy, cross-modal alignment, and model interpretability. Code is available at https://github.com/AI-14/micar-vl-moe.
Paper Structure (24 sections, 7 equations, 8 figures, 3 tables)

This paper contains 24 sections, 7 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision Language Models
  4. Multiscale Image Feature Extraction
  5. Multimodal Alignment Mechanisms
  6. Mixture of Expert Models
  7. Method
  8. Problem Formulation
  9. Overview of the Approach
  10. Multiscale Vision Encoder
  11. Latent Encoder
  12. Report Embedding
  13. Latent MoE Decoder
  14. Loss Function
  15. Experiments
  16. ...and 9 more sections

Figures (8)

  • Figure 1: An overview of the proposed MicarVLMoE model highlighting the overall architecture for medical image reporting with core modules encompassing -- multiscale vision encoder, latent encoder, and latent MoE decoder with gated fusion.
  • Figure 2: Illustration of multiscale vision encoder mechanism.
  • Figure 3: Visualization of hierarchical feature maps of the MSVE module.
  • Figure 4: Visualization of multihead dual-branch latent attention mechanism. In the figure, the matrix size notations are as follows: $b$: batch size; $s$: sequence length; $d_{m}$: $d_{\text{model}}$; $d_l$: $d_{\text{latent}}$; $h$: number of heads; $d_{q_n}$,$d_{q_r}$,$d_{k_n}$,$d_{k_r}$: latent dimension of normal and rope query and key.
  • Figure 5: Example of generated reports compared between SAT, Transformer, and our (MicarVLMoE) model. Highlighted colors depict medically relevant findings matched with the ground truth reports/captions.
  • ...and 3 more figures