Accurate and Scalable Multimodal Pathology Retrieval via Attentive Vision-Language Alignment

TL;DR

PathSearch is a scalable multimodal retrieval framework for whole-slide pathology images that unifies fine-grained attentive mosaics with global slide embeddings through vision–language contrastive learning. Trained on 6,926 slide–report pairs from TCGA, it supports image-to-image, image-to-text, and text-to-text retrieval, achieving state-of-the-art performance across seven datasets and improving pathologists’ diagnostic accuracy in a reader study. The method delivers robust cross-institutional generalization and linear scalability to large archives by using a fixed mosaic representation (e.g., $M=16$), while retaining morphologically rich, interpretable retrieval. These results highlight PathSearch as a practical, retrieval-augmented tool for clinical decision support, education, and research in digital pathology, with potential to underpin future AI-assisted diagnostic workflows.

Abstract

The rapid digitization of histopathology slides has opened up new possibilities for computational tools in clinical and research workflows. Among these, content-based slide retrieval stands out, enabling pathologists to identify morphologically and semantically similar cases, thereby supporting precise diagnoses, enhancing consistency across observers, and assisting example-based education. However, effective retrieval of whole slide images (WSIs) remains challenging due to their gigapixel scale and the difficulty of capturing subtle semantic differences amid abundant irrelevant content. To overcome these challenges, we present PathSearch, a retrieval framework that unifies fine-grained attentive mosaic representations with global-wise slide embeddings aligned through vision-language contrastive learning. Trained on a corpus of 6,926 slide-report pairs, PathSearch captures both fine-grained morphological cues and high-level semantic patterns to enable accurate and flexible retrieval. The framework supports two key functionalities: (1) mosaic-based image-to-image retrieval, ensuring accurate and efficient slide research; and (2) multi-modal retrieval, where text queries can directly retrieve relevant slides. PathSearch was rigorously evaluated on four public pathology datasets and three in-house cohorts, covering tasks including anatomical site retrieval, tumor subtyping, tumor vs. non-tumor discrimination, and grading across diverse organs such as breast, lung, kidney, liver, and stomach. External results show that PathSearch outperforms traditional image-to-image retrieval frameworks. A multi-center reader study further demonstrates that PathSearch improves diagnostic accuracy, boosts confidence, and enhances inter-observer agreement among pathologists in real clinical scenarios. These results establish PathSearch as a scalable and generalizable retrieval solution for digital pathology.

Figures (10)

• Figure 1: Overview of the PathSearch framework and retrieval pipeline.(a) Clinical motivation: an unseen whole-slide image (WSI) is embedded and compared against a database of historical cases, enabling retrieval of precedent slides and reports to support precision diagnosis. (b) Composition of the training corpus, consisting of 6,926 slide–report pairs from TCGA across diverse organs. (c) Core training structure of PathSearch. For the vision branch, a WSI is tiled into patches and encoded into patch embeddings ($E_p$), followed by constructing multi-grained slide representations, which includes an attentive mosaic encoder ($E_m$) for fine-grained features and a global-wise semantic encoder ($E_s$) for slide-level features. For the text branch, a LLM is first used to process raw pathology reports into structured text expressions. Then, a learnable text encoder ($E_t$) is used to generates text embeddings ($E_t$) based on the formatted text inputs. Finally, $E_s$ and $E_t$ are aligned within a shared space through contrastive learning. (d) Image-query retrieval. The multi-grained image embedding supports two modes: mosaic-based image-to-image retrieval for morphological similarity and slide embedding-based image-to-text retrieval for conceptual similarity. (e) Text-query retrieval. Text embeddings ($E_t$) enable semantic retrieval of both related slides and reports from the database.
• Figure 2: Comprehensive performance evaluation of PathSearch against state-of-the-art methods.(a) A radar plot of the comparative image-to-image Top-1 retrieval accuracy across all seven validation datasets. PathSearch consistently achieves the highest performance across diverse clinical tasks and organ types. (b, c) Comparison of multi-modal retrieval performance on Image-to-Image (I2I), Image-to-Text (I2T), Text-to-Image (T2I), and Text-to-Text (T2T) tasks. The results demonstrate the significant performance boost provided by PathSearch's attentive mosaic mechanism and the vision-language alignment. (d-i) Detailed retrieval performance on the six external datasets: (d) Camelyon16, (e) Camelyon17, (f) DHMC-LUAD, (g) FAHZJU Gastric Grading, (h) SRRSH HCC Grading, and (i) FAHWMU HCC Risk Evaluation.
• Figure 3: Qualitative examples of retrieval on the TCGA test set comparing PathSearch, CHIEF, and SISH. Three query slides and one text query are shown alongside their top-5 retrieved results. Green checkmarks (✓) denote correct retrievals, while red crosses ($\times$) denote incorrect retrievals. (Top row) A LUSC query slide retrieves five related LUSC slides, exemplifying reliable support for subtype-level diagnosis. (Second row) A KIRP query slide with manual markings correctly retrieves five KIRP slides. In contrast, CHIEF mistakenly retrieves an irrelevant LUSC slide, while SISH performs correctly, highlighting the robustness of mosaic-based approaches to noise. The right side holds the slide embeddings distribution of PathSearch, which shows great separability. (Third row) A BRCA-IDC query slide correctly retrieves four IDC slides but also returns one ILC slide at rank 5. In comparison, CHIEF retrieves an irrelevant LUSC slide, and SISH retrieves three incorrect ILC slides, demonstrating PathSearch's superior semantic understanding of IDC cases. The right side resides the slide embeddings distribution of CHIEF, which shows moderate separability. (Bottom row) A text-based query describing a LUSC case retrieves four correct LUSC slides and one LUAD slide at rank 2, reflecting the known visual and semantic similarity between these NSCLC subtypes. Together, these examples illustrate the accuracy and interpretability of PathSearch against state-of-the-art methods.
• Figure 4: Qualitative examples of PathSearch's retrieval on the Camelyon17 dataset. The figure compares the retrieval performance of PathSearch against two state-of-the-art methods, CHIEF and SISH. Green checkmarks (✓) indicate a correct retrieval (label matches the query), while red crosses ($\times$) indicate an incorrect retrieval. (Top Row) For a straightforward non-tumor query, all three methods perform perfectly, demonstrating baseline competence. (Second Row) For a representative tumor query, PathSearch correctly identifies five morphologically similar tumor cases. In stark contrast, CHIEF and SISH both make multiple errors, retrieving several non-tumor slides and highlighting PathSearch's superior discriminative power. (Third Row) In a challenging non-tumor query, PathSearch makes only one mistake at rank 3. CHIEF's performance degrades significantly, with four incorrect retrievals, while SISH also makes one error. (Bottom Row) For a difficult tumor query, PathSearch again performs perfectly. In comparison, both CHIEF and SISH struggle, each making critical errors in their top-ranked results. Collectively, these examples demonstrate PathSearch's consistently higher accuracy and robustness, especially in challenging cases, enhancing its reliability for retrieving clinically relevant precedents.
• Figure 5: Computational complexity analysis of PathSearch compared to traditional mosaic-based methods. The scalability of retrieval systems is analyzed with respect to database size and image resolution. (a) A log-scale plot illustrates the relative computational cost as a function of the number of slides in the database, denoted by $S$. PathSearch's retrieval cost exhibits linear scalability ($O(S)$). In contrast, the cost for traditional methods (e.g., Yottixel, SISH) scales quadratically with the number of patches per slide, denoted by $P$, resulting in prohibitive computational demands as $P$ increases. The different lines for traditional methods show this effect for various patch sampling percentages $f$ (5%, 10%, 15%) on slides with different total patch counts ($P$ = 1000, 5000, and 20000). (b) A log-scale plot shows the maximum supportable database size under a fixed computational budget as a function of the average number of patches per slide ($P$). PathSearch's scalability is independent of slide resolution, as it uses a fixed-size mosaic. Conversely, the capacity of traditional methods diminishes exponentially as slide resolution increases, severely limiting their applicability to large-scale, high-resolution archives.