ChatEXAONEPath: An Expert-level Multimodal Large Language Model for Histopathology Using Whole Slide Images

TL;DR

ChatEXAONEPath tackles the need for WSIs-level, expert-level multimodal reasoning in histopathology by integrating a specialized WSI vision tower with a CLAM-based patch aggregator, a genetics-aware aggregator, and an LLM backbone fine-tuned via a two-phase process. It introduces RAIDER, a retrieval-augmented data generation pipeline that scales WSIs-text pairs from TCGA to train and evaluate a powerful histopathology assistant, leveraging both GPT-4o and open-source LLaMA3.1-based systems. An AI-based evaluation protocol with seven criteria provides a structured, albeit imperfect, mechanism to adjudicate generated diagnoses, revealing both promising acceptance rates (62.9% reported in the abstract and applicable to test datasets) and limitations in evaluator reliability. The work demonstrates pan-cancer WSI understanding and the potential clinical utility of integrated multimodal reasoning while identifying key challenges in data quality, alignment, and evaluation that warrant further multimodal, human-aligned validation.

Abstract

Recent studies have made significant progress in developing large language models (LLMs) in the medical domain, which can answer expert-level questions and demonstrate the potential to assist clinicians in real-world clinical scenarios. Studies have also witnessed the importance of integrating various modalities with the existing LLMs for a better understanding of complex clinical contexts, which are innately multi-faceted by nature. Although studies have demonstrated the ability of multimodal LLMs in histopathology to answer questions from given images, they lack in understanding of thorough clinical context due to the patch-level data with limited information from public datasets. Thus, developing WSI-level MLLMs is significant in terms of the scalability and applicability of MLLMs in histopathology. In this study, we introduce an expert-level MLLM for histopathology using WSIs, dubbed as ChatEXAONEPath. We present a retrieval-based data generation pipeline using 10,094 pairs of WSIs and histopathology reports from The Cancer Genome Atlas (TCGA). We also showcase an AI-based evaluation protocol for a comprehensive understanding of the medical context from given multimodal information and evaluate generated answers compared to the original histopathology reports. We demonstrate the ability of diagnosing the given histopathology images using ChatEXAONEPath with the acceptance rate of 62.9% from 1,134 pairs of WSIs and reports. Our proposed model can understand pan-cancer WSIs and clinical context from various cancer types. We argue that our proposed model has the potential to assist clinicians by comprehensively understanding complex morphology of WSIs for cancer diagnosis through the integration of multiple modalities.

Figures (7)

• Figure 1: A qualitative result and evaluation from the test dataset. ChatEXAONEPath answered from the given whole slide image and question. The evaluator AI model generated reasons to accept the best answer, and made a final decision.
• Figure 2: The overview of the structure of ChatEXAONEPath for training phase 1, vision-language alignment of whole slide images and corresponding captions.
• Figure 3: The overview of the structure of ChatEXAONEPath for training phase 2, instruction tuning of LLaMA2:7B model for generating answers from the given input question and the input whole slide image. Compared to the phase 1, the weights of linear projection layers are fine-tuned using low-rank adaptation technique (LoRA).
• Figure 4: Overview of AI-based evaluation process. After then end of the training, we generated 10 answers per each question, which are then evaluated by the evaluator LLM to make a decision of acceptance or rejection.
• Figure A1: A qualitative result and evaluation from the test dataset. ChatEXAONEPath-v3 answered from the given whole slide image and question. The evaluator AI model generated reasons to accept the best answer, and made a final decision.