Democratizing Pathology Co-Pilots: An Open Pipeline and Dataset for Whole-Slide Vision-Language Modelling

TL;DR

Open problems in pathology VLMs include limited reproducibility and lack of true WSI-level understanding. The authors introduce Polysome for synthetic instruction generation, HISTAI-Instruct as a large open WSI-instruction dataset, and ANTONI-α, a WSI-focused vision-language model trained on HISTAI-Instruct. They demonstrate that domain-specific, high-resolution processing and scalable instruction-tuning improve organ identification, neoplasm detection, and differential diagnosis compared with public baselines, while releasing code and data to support reproducibility. This work advances transparent, scalable, and interactive digital pathology through an open, end-to-end pipeline.

Abstract

Vision-language models (VLMs) have the potential to become co-pilots for pathologists. However, most VLMs either focus on small regions of interest within whole-slide images, provide only static slide-level outputs, or rely on data that is not publicly available, limiting reproducibility. Furthermore, training data containing WSIs paired with detailed clinical reports is scarce, restricting progress toward transparent and generalisable VLMs. We address these limitations with three main contributions. First, we introduce Polysome, a standardised tool for synthetic instruction generation. Second, we apply Polysome to the public HISTAI dataset, generating HISTAI-Instruct, a large whole-slide instruction tuning dataset spanning 24,259 slides and over 1.1 million instruction-response pairs. Finally, we use HISTAI-Instruct to train ANTONI-α, a VLM capable of visual-question answering (VQA). We show that ANTONI-α outperforms MedGemma on WSI-level VQA tasks of tissue identification, neoplasm detection, and differential diagnosis. We also compare the performance of multiple incarnations of ANTONI-α trained with different amounts of data. All methods, data, and code are publicly available.

Figures (4)

• Figure 1: HISTAI data preprocessing pipeline. Blue: retained cases; orange: discarded cases.
• Figure 2: Architecture of ANTONI-$\alpha$. Image processing modules (blue) extract features via VIRCHOW and PRISM. These features are aligned with conversational data (green) via a Vision Projector. The MedGemma LLM (pink) generates responses using inputs from both modalities. Snowflake and flame icons denote frozen and trainable parameters, respectively, during the instruction-tuning stage (in contrast to pretraining, where the LLM is fully frozen).
• Figure 3: Validation pipeline for comparing ANTONI-$\alpha$ and MedGemma. Both models process the same WSI. For MedGemma, the WSI is first downscaled and packed. The evaluation consists of three questions: Q1 targets organ or tissue identification, Q2 detects the presence of a neoplasm, and Q3 requires the most likely diagnosis selected from three possible candidate differentials.
• Figure 4: Qualitative comparison of ANTONI-$\alpha$ and MedGemma on a dermatology case (dermatofibroma). ANTONI-$\alpha$ (left) processes the full-resolution WSI and synthesizes its findings into the correct diagnosis of dermatofibroma. In contrast, MedGemma (right) relies on a lower resolution thumbnail. It is unable to assess margins or describe cell details, leading to an incorrect diagnosis.