Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Basit Alawode, Arif Mahmood, Muaz Khalifa Al-Radi, Shahad Albastaki, Asim Khan, Muhammad Bilal, Moshira Ali Abdalla, Mohammed Bennamoun, Sajid Javed

Abstract

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce \textbf{MLLM-HWSI}, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight \textit{Cell-Cell Attention Fusion (CCAF)} transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: \href{https://github.com/BasitAlawode/HWSI-MLLM}{GitHub}.

MLLM-HWSI: A Multimodal Large Language Model for Hierarchical Whole Slide Image Understanding

Abstract

Whole Slide Images (WSIs) exhibit hierarchical structure, where diagnostic information emerges from cellular morphology, regional tissue organization, and global context. Existing Computational Pathology (CPath) Multimodal Large Language Models (MLLMs) typically compress an entire WSI into a single embedding, which hinders fine-grained grounding and ignores how pathologists synthesize evidence across different scales. We introduce \textbf{MLLM-HWSI}, a Hierarchical WSI-level MLLM that aligns visual features with pathology language at four distinct scales, cell as word, patch as phrase, region as sentence, and WSI as paragraph to support interpretable evidence-grounded reasoning. MLLM-HWSI decomposes each WSI into multi-scale embeddings with scale-specific projectors and jointly enforces (i) a hierarchical contrastive objective and (ii) a cross-scale consistency loss, preserving semantic coherence from cells to the WSI. We compute diagnostically relevant patches and aggregate segmented cell embeddings into a compact cellular token per-patch using a lightweight \textit{Cell-Cell Attention Fusion (CCAF)} transformer. The projected multi-scale tokens are fused with text tokens and fed to an instruction-tuned LLM for open-ended reasoning, VQA, report, and caption generation tasks. Trained in three stages, MLLM-HWSI achieves new SOTA results on 13 WSI-level benchmarks across six CPath tasks. By aligning language with multi-scale visual evidence, MLLM-HWSI provides accurate, interpretable outputs that mirror diagnostic workflows and advance holistic WSI understanding. Code is available at: \href{https://github.com/BasitAlawode/HWSI-MLLM}{GitHub}.
Paper Structure (30 sections, 9 equations, 5 figures, 20 tables)

This paper contains 30 sections, 9 equations, 5 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Proposed Hierarchical WSI MLLM
  4. Hierarchical Decomposition of Gigapixel WSIs
  5. Architecture
  6. Hierarchical Multi-Scale Encoder
  7. Hierarchical Alignment (V $\rightarrow$ L) Projectors
  8. Multimodal Large Language Model (LLM)
  9. Training Strategy
  10. Experiments
  11. Evaluation Metrics and SOTA Comparisons
  12. Ablation Studies
  13. Main Results
  14. Acknowledgement
  15. Inference Details
  16. ...and 15 more sections

Figures (5)

  • Figure 1: Our proposed MLLM-HWSI model aligns WSIs across multiple scales e.g., cells, patches, regions, and WSI enabling fine-grained, context-aware, and interpretable pathology reasoning.
  • Figure 2: Comparison of MLLM-HWSI with SOTA methods.
  • Figure 3: Overview of the proposed MLLM-HWSI. (A) Hierarchical decomposition of WSI into cell, patch, and region-level embeddings aligned with MLLM. (B) MLLM-HWSI three stage pre-training paradigm for multimodal reasoning.
  • Figure 4: Performance comparison of the proposed MLLM-HWSI with SOTA CPath models. MLLM-HWSI achieves the highest overall scores across all benchmarks, underscoring the benefits of hierarchical multi-scale visual encoding and cross-modal alignment.
  • Figure 5: Pathologically meaningful patches and discarded patch.