Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deep Learning-based Prediction of Breast Cancer Tumor and Immune Phenotypes from Histopathology

Tiago Gonçalves, Dagoberto Pulido-Arias, Julian Willett, Katharina V. Hoebel, Mason Cleveland, Syed Rakin Ahmed, Elizabeth Gerstner, Jayashree Kalpathy-Cramer, Jaime S. Cardoso, Christopher P. Bridge, Albert E. Kim

TL;DR

The paper tackles the problem of deriving reproducible tumor microenvironment phenotypes for individual breast cancer patients by predicting pathway activity from hematoxylin and eosin slides using MIL-based deep learning. It compares CLAM and TransMIL architectures with two feature extractors (ResNet50/ImageNet and PLIP/OpenPath) and uses ssGSEA-derived binary pathway labels to train models. The results show AUROCs above 0.70 for most pathways and 0.75–0.80 for several immune-related programs, with PLIP features and attention maps supporting biologically meaningful learning. This work demonstrates the feasibility of computational H&E biomarkers for precision oncology and provides a publicly available pipeline with directions for multi-modal extensions.

Abstract

The interactions between tumor cells and the tumor microenvironment (TME) dictate therapeutic efficacy of radiation and many systemic therapies in breast cancer. However, to date, there is not a widely available method to reproducibly measure tumor and immune phenotypes for each patient's tumor. Given this unmet clinical need, we applied multiple instance learning (MIL) algorithms to assess activity of ten biologically relevant pathways from the hematoxylin and eosin (H&E) slide of primary breast tumors. We employed different feature extraction approaches and state-of-the-art model architectures. Using binary classification, our models attained area under the receiver operating characteristic (AUROC) scores above 0.70 for nearly all gene expression pathways and on some cases, exceeded 0.80. Attention maps suggest that our trained models recognize biologically relevant spatial patterns of cell sub-populations from H&E. These efforts represent a first step towards developing computational H&E biomarkers that reflect facets of the TME and hold promise for augmenting precision oncology.

Deep Learning-based Prediction of Breast Cancer Tumor and Immune Phenotypes from Histopathology

TL;DR

The paper tackles the problem of deriving reproducible tumor microenvironment phenotypes for individual breast cancer patients by predicting pathway activity from hematoxylin and eosin slides using MIL-based deep learning. It compares CLAM and TransMIL architectures with two feature extractors (ResNet50/ImageNet and PLIP/OpenPath) and uses ssGSEA-derived binary pathway labels to train models. The results show AUROCs above 0.70 for most pathways and 0.75–0.80 for several immune-related programs, with PLIP features and attention maps supporting biologically meaningful learning. This work demonstrates the feasibility of computational H&E biomarkers for precision oncology and provides a publicly available pipeline with directions for multi-modal extensions.

Abstract

The interactions between tumor cells and the tumor microenvironment (TME) dictate therapeutic efficacy of radiation and many systemic therapies in breast cancer. However, to date, there is not a widely available method to reproducibly measure tumor and immune phenotypes for each patient's tumor. Given this unmet clinical need, we applied multiple instance learning (MIL) algorithms to assess activity of ten biologically relevant pathways from the hematoxylin and eosin (H&E) slide of primary breast tumors. We employed different feature extraction approaches and state-of-the-art model architectures. Using binary classification, our models attained area under the receiver operating characteristic (AUROC) scores above 0.70 for nearly all gene expression pathways and on some cases, exceeded 0.80. Attention maps suggest that our trained models recognize biologically relevant spatial patterns of cell sub-populations from H&E. These efforts represent a first step towards developing computational H&E biomarkers that reflect facets of the TME and hold promise for augmenting precision oncology.
Paper Structure (11 sections, 2 figures, 2 tables)

This paper contains 11 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. WSI Data Preparation
  5. Quantification of Gene Pathway Expression
  6. Models and Training Details
  7. CLAM
  8. TransMIL
  9. Results and Discussion
  10. Conclusion and Future Work
  11. Acknowledgments

Figures (2)

  • Figure 1: Overview of the proposed pipeline.
  • Figure 2: Attention map obtained using the AM-SB architecture for a WSI predicted to have a high-degree of T-cell mediated cytotoxicity. Red zones and blue zones indicate high and low attention scores, respectively. On the right, we provide two exemplar patches: 1) the high-attention patch illustrates abundant tumor-infiltrating lymphocytes without tumor cells, which are suggestive of high immune activity, and 2) the low-attention region demonstrated areas of tumor necrosis and minimal lymphocytes, consistent with low immune activity.