GMAT: Grounded Multi-Agent Clinical Description Generation for Text Encoder in Vision-Language MIL for Whole Slide Image Classification

TL;DR

The paper addresses the limited expressiveness and domain grounding of prompts in vision-language MIL for whole-slide image classification by introducing GMAT, a grounded multi-agent text generation framework. GMATG leverages a structured knowledge base from pathology textbooks and a planning/generation/verification/finalization workflow to produce diverse, clinically grounded per-class descriptions; these descriptions are encoded as a list and aligned with multi-scale visual features through a CONCH-based MIL pipeline. The approach yields improvements in zero-shot and competitive performance in fine-tuned settings on TCGA-RCC and TCGA-Lung, with ablations confirming the benefit of the multi-agent design. Overall, GMAT enhances interpretability and accuracy in computational pathology by enriching prompts with domain-informed, collaborative descriptions.

Abstract

Multiple Instance Learning (MIL) is the leading approach for whole slide image (WSI) classification, enabling efficient analysis of gigapixel pathology slides. Recent work has introduced vision-language models (VLMs) into MIL pipelines to incorporate medical knowledge through text-based class descriptions rather than simple class names. However, when these methods rely on large language models (LLMs) to generate clinical descriptions or use fixed-length prompts to represent complex pathology concepts, the limited token capacity of VLMs often constrains the expressiveness and richness of the encoded class information. Additionally, descriptions generated solely by LLMs may lack domain grounding and fine-grained medical specificity, leading to suboptimal alignment with visual features. To address these challenges, we propose a vision-language MIL framework with two key contributions: (1) A grounded multi-agent description generation system that leverages curated pathology textbooks and agent specialization (e.g., morphology, spatial context) to produce accurate and diverse clinical descriptions; (2) A text encoding strategy using a list of descriptions rather than a single prompt, capturing fine-grained and complementary clinical signals for better alignment with visual features. Integrated into a VLM-MIL pipeline, our approach shows improved performance over single-prompt class baselines and achieves results comparable to state-of-the-art models, as demonstrated on renal and lung cancer datasets.

Figures (2)

• Figure 1: Overview of GMATG. A team of specialized agents generates class-specific descriptions from domain knowledge, covering morphological, molecular, and clinical aspects. These are combined into rich text embeddings to guide visual understanding.
• Figure 2: Overview of our MIL framework for 5× magnification. Patch features at 5× magnification are embedded using the CONCH visual encoder and matched with GMAT-generated text descriptions. Similarity scores are computed via visual-text alignment and aggregated using soft attention to produce class logits. Features from 10× magnification are processed in parallel, and predictions from both scales are fused for final slide-level classification.