Going Beyond H&E and Oncology: How Do Histopathology Foundation Models Perform for Multi-stain IHC and Immunology?

TL;DR

This study evaluates the generalisation capabilities of state-of-the-art histopathology foundation models on out-of-distribution multi-stain autoimmune Immunohistochemistry datasets and findings highlight the challenges in transferring knowledge from cancer to autoimmune histopathology.

Abstract

This study evaluates the generalisation capabilities of state-of-the-art histopathology foundation models on out-of-distribution multi-stain autoimmune Immunohistochemistry datasets. We compare 13 feature extractor models, including ImageNet-pretrained networks, and histopathology foundation models trained on both public and proprietary data, on Rheumatoid Arthritis subtyping and Sjogren's Disease detection tasks. Using a simple Attention-Based Multiple Instance Learning classifier, we assess the transferability of learned representations from cancer H&E images to autoimmune IHC images. Contrary to expectations, histopathology-pretrained models did not significantly outperform ImageNet-pretrained models. Furthermore, there was evidence of both autoimmune feature misinterpretation and biased feature importance. Our findings highlight the challenges in transferring knowledge from cancer to autoimmune histopathology and emphasise the need for careful evaluation of AI models across diverse histopathological tasks. The code to run this benchmark is available at https://github.com/AmayaGS/ImmunoHistoBench.

Figures (15)

• Figure 1: Differences in histological patterns between cancer and autoimmune pathologies. A-C: IHC staining, D: H&E staining. A: Breast cancer tumours (A1), heavily infiltrated with immune cells throughout (A2), B: Breast cancer tumour (B1), characterised by TMEs (B2). Tumours with immune infiltrate show better prognosis. C: Tissue with strong presence of immune infiltrates in autoimmune disease (C1) such as ELS (C2), which can correlate to poor prognosis. Note A2 shows good prognosis, while C2 shows poor. D: differences in H&E staining between cancer (D1) and autoimmune (D2). In D1, we see examples of TMEs surrounded by more darkly stained immune cells unable to infiltrate the tumour. Cancer cells show enlarged cytoplasm to nuclear ratio. D2 shows darkly stained immune cells.
• Figure 2: Schematic representation of the multi-stain whole slide image (WSI) analysis pipeline. The process encompasses (1) tissue thresholding and patch extraction, (2) feature extraction yielding patient-level feature matrices, and (3) attention-based multiple instance learning (ABMIL) classification. This study primarily focuses on comparing feature extraction methodologies.
• Figure 3: Rheumatoid Arthritis inflammatory pathotypes based on semi-quantitative analysis of synovial tissue biopsies stained with H&E, CD20+ B cells, CD68+ macrophages and IHC+ CD138 plasma cells Humby2021AGS23_BMVC.
• Figure 4: Example of sicca vs Sjogren's Disease presentation in H&E and IHC stains. On top, a patient diagnosed with sicca, on bottom a patient diagnosed with Sjogren's disease. Here we show samples stained with IHC stains CD3+ T cells, CD20+ B cells and CD138+ plasma cells.
• Figure 5: Performance comparison of feature extractor models on Rheumatoid Arthritis Subtyping. ImageNet pretrained models are shown in orange, Histopathology pretrained models in blue for models trained on publicly available datasets and green for proprietary data. Within each category and metric models are organised from most to least performant, with the top three performers highlighted with a red box.