Mew: Multiplexed Immunofluorescence Image Analysis through an Efficient Multiplex Network

TL;DR

Mew tackles two critical bottlenecks in multiplexed immunofluorescence image analysis—cellular heterogeneity and scalability—by modeling mIF data as a two-layer multiplex network (Voronoi geometry and cell-type connectivity) processed with a scalable, precomputed GNN. It introduces a stochastic edge sampling strategy for the cell-type layer and a Voronoi-Cell-type Attention module to automatically weigh the contribution of each network for patient-level phenotype predictions, including binary outcomes and hazard modeling. Across multi-institution datasets, Mew achieves state-of-the-art performance and demonstrates strong generalization and superior efficiency, with substantial reductions in preprocessing and evaluation times. The approach offers a practical and interpretable framework for p hysiology-informed mIF analysis and can be extended with multi-modal data or applied to WSIs, potentially accelerating clinically relevant insights from tissue imaging.

Abstract

Recent advancements in graph-based approaches for multiplexed immunofluorescence (mIF) images have significantly propelled the field forward, offering deeper insights into patient-level phenotyping. However, current graph-based methodologies encounter two primary challenges: (1) Cellular Heterogeneity, where existing approaches fail to adequately address the inductive biases inherent in graphs, particularly the homophily characteristic observed in cellular connectivity and; (2) Scalability, where handling cellular graphs from high-dimensional images faces difficulties in managing a high number of cells. To overcome these limitations, we introduce Mew, a novel framework designed to efficiently process mIF images through the lens of multiplex network. Mew innovatively constructs a multiplex network comprising two distinct layers: a Voronoi network for geometric information and a Cell-type network for capturing cell-wise homogeneity. This framework equips a scalable and efficient Graph Neural Network (GNN), capable of processing the entire graph during training. Furthermore, Mew integrates an interpretable attention module that autonomously identifies relevant layers for image classification. Extensive experiments on a real-world patient dataset from various institutions highlight Mew's remarkable efficacy and efficiency, marking a significant advancement in mIF image analysis. The source code of Mew can be found here: \url{https://github.com/UNITES-Lab/Mew}

Figures (9)

• Figure 1: mIF image.
• Figure 2: Distribution of mIF image is examined through (a) Cellular Heterogeneity: Homophily Ratio and (b) Scalability: number of cells with (c) case study of mIF cellular graphs in the UPMC dataset. Homophily Ratio: $\frac{\# \text{of edges connecting same cell-type nodes}}{\# \text{of total edges}}$. Distributions on other datasets are provided in Appendix \ref{['appendix:A']}.
• Figure 3: Overall framework of Mew. Given mIF images, it employs Delaunay triangulation and cell segmentation, leading to the formation of a multiplex network composed of two distinct networks: a Voronoi network and a Cell-type network. Here, two networks share common nodes but have distinctive edge connections. These networks are then analyzed using scalable GNNs equipped with precomputing capabilities and stochastic edge sampling techniques. This analysis is further enhanced by the Voronoi-Cell-type Attention, a mechanism designed for evaluating the significance of each layer. Ultimately, it predicts the patient's phenotype via the phenotype prediction head.
• Figure 4: Illustration of Precomputing and Training procedure. Once the orange colored components, ${\mathbf{X}^{I}}$, ${\mathbf{A}^{I}_{(1)}\mathbf{X}^{1}}$, $\cdots$, ${\mathbf{A}^{I}_{(K)}\mathbf{X}^{I}}$ are precomputed, they are utilized throughout the training and inference phases.
• Figure 5: Ablation study of Mew on the Stanford-CNC dataset for (a) Binary Classification (b) Hazard Modeling. Here, $\mathcal{G}$, $\mathcal{G'}$, and $\tilde{\mathcal{G}}$ denote the Voronoi network, Cell-type network, and Multiplex network, respectively. 'S' indicates the Stochastic Edge Sampling technique, while '$\oplus$' and '$||$' represent addition and concatenation operations, respectively. Applying the attention mechanism to the multiplex network completes Mew.