Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IHC Matters: Incorporating IHC analysis to H&E Whole Slide Image Analysis for Improved Cancer Grading via Two-stage Multimodal Bilinear Pooling Fusion

Jun Wang, Yu Mao, Yufei Cui, Nan Guan, Chun Jason Xue

TL;DR

This work tackles improving cancer grading by leveraging immunohistochemistry (IHC) alongside hematoxylin and eosin (H&E) whole-slide images (WSIs). It introduces a two-stage multimodal bilinear pooling framework with an attention-based MIL pipeline and a fusion module that uses Bilinear Average Pooling and the Kronecker product to fuse H&E and IHC features, producing robust joint representations. On public breast cancer datasets, the approach achieves state-of-the-art performance (e.g., AUC ≈ 0.996 and ACC ≈ 0.953 on BCI), and demonstrates superior accuracy over single-modality baselines on IHC4BC as well, while showing that pixel-level alignment between H&E and IHC is not necessary. These results underscore the practical potential of multimodal histopathology for accurate cancer grading and suggest broad applicability to other histology tasks.

Abstract

Immunohistochemistry (IHC) plays a crucial role in pathology as it detects the over-expression of protein in tissue samples. However, there are still fewer machine learning model studies on IHC's impact on accurate cancer grading. We discovered that IHC and H\&E possess distinct advantages and disadvantages while possessing certain complementary qualities. Building on this observation, we developed a two-stage multi-modal bilinear model with a feature pooling module. This model aims to maximize the potential of both IHC and HE's feature representation, resulting in improved performance compared to their individual use. Our experiments demonstrate that incorporating IHC data into machine learning models, alongside H\&E stained images, leads to superior predictive results for cancer grading. The proposed framework achieves an impressive ACC higher of 0.953 on the public dataset BCI.

IHC Matters: Incorporating IHC analysis to H&E Whole Slide Image Analysis for Improved Cancer Grading via Two-stage Multimodal Bilinear Pooling Fusion

TL;DR

This work tackles improving cancer grading by leveraging immunohistochemistry (IHC) alongside hematoxylin and eosin (H&E) whole-slide images (WSIs). It introduces a two-stage multimodal bilinear pooling framework with an attention-based MIL pipeline and a fusion module that uses Bilinear Average Pooling and the Kronecker product to fuse H&E and IHC features, producing robust joint representations. On public breast cancer datasets, the approach achieves state-of-the-art performance (e.g., AUC ≈ 0.996 and ACC ≈ 0.953 on BCI), and demonstrates superior accuracy over single-modality baselines on IHC4BC as well, while showing that pixel-level alignment between H&E and IHC is not necessary. These results underscore the practical potential of multimodal histopathology for accurate cancer grading and suggest broad applicability to other histology tasks.

Abstract

Immunohistochemistry (IHC) plays a crucial role in pathology as it detects the over-expression of protein in tissue samples. However, there are still fewer machine learning model studies on IHC's impact on accurate cancer grading. We discovered that IHC and H\&E possess distinct advantages and disadvantages while possessing certain complementary qualities. Building on this observation, we developed a two-stage multi-modal bilinear model with a feature pooling module. This model aims to maximize the potential of both IHC and HE's feature representation, resulting in improved performance compared to their individual use. Our experiments demonstrate that incorporating IHC data into machine learning models, alongside H\&E stained images, leads to superior predictive results for cancer grading. The proposed framework achieves an impressive ACC higher of 0.953 on the public dataset BCI.
Paper Structure (21 sections, 7 equations, 2 figures, 2 tables)

This paper contains 21 sections, 7 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. IHC
  4. Multimodal
  5. Method
  6. Two-stage Multimodal bilinear pooling framework
  7. Attention-based multiple instance learning training pipeline
  8. The training process for a single branch
  9. Multi-branch multimodal bilinear framework
  10. Loss
  11. Fusion Module
  12. Bilinear Average Pooling
  13. Kronecker product
  14. Experiments
  15. Datasets
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Our framework: two-stage multimodal bilinear pooling Framework
  • Figure 2: H&E-based machine learning the wrong prediction. However, for their paired IHC images, there are huge differences in color and can be identified correctly in the IHC-training pipeline.