Fluoroformer: Scaling multiple instance learning to multiplexed images via attention-based channel fusion

TL;DR

The Fluoroformer module is presented, specifically tailored to multiplexed WSIs by leveraging scaled dot-product attention (SDPA) to interpretably fuse information across disparate channels and provides a path for adapting state-of-the-art AI techniques to emerging spatial biology assays.

Abstract

Though multiple instance learning (MIL) has been a foundational strategy in computational pathology for processing whole slide images (WSIs), current approaches are designed for traditional hematoxylin and eosin (H&E) slides rather than emerging multiplexed technologies. Here, we present an MIL strategy, the Fluoroformer module, that is specifically tailored to multiplexed WSIs by leveraging scaled dot-product attention (SDPA) to interpretably fuse information across disparate channels. On a cohort of 434 non-small cell lung cancer (NSCLC) samples, we show that the Fluoroformer both obtains strong prognostic performance and recapitulates immuno-oncological hallmarks of NSCLC. Our technique thereby provides a path for adapting state-of-the-art AI techniques to emerging spatial biology assays.

Figures (7)

• Figure 1: Overview of the Fluoroformer strategy for multiplexed imaging. Mathematical symbols are defined in text.
• Figure 2: Heatmaps of the most highly attended markers in each patch of selected mIF images for the Fluoroformer model with a ResNet50 embedder. Patterns are observed such as DAPI being a highly attended marker for alveolar tissue (A, B) and CD8-attended regions appearing at tumor margins and sporadically within the tumor (A, C, D).
• Figure 3: Heatmaps of patch attention for each combination of imaging modality and neural embedder. The mIF heatmaps often demonstrate higher spatial continuity and are concentrated towards regions adjacent to the tumor mass (e.g., B, C), whereas the H&E maps are often sparser and concentrate on inner tumor regions.
• Figure 4: Average marker attention matrix ($\mathbf A_k$) across the cohort for the Fluoroformer with ResNet embeddings. The displayed values are $z$-scored, meaning a value of $1$ indicates that the mean attention value for that entry is one standard deviation higher than the global mean across all markers. The $x$- and $y$-axes represent in- and outgoing attention, respectively.
• Figure 5: Heatmaps of patch attention for each combination of imaging modality and neural embedder. The data is the same as displayed in the main text, but at high resolution.