Cross-modal ultra-scale learning with tri-modalities of renal biopsy images for glomerular multi-disease auxiliary diagnosis

TL;DR

This work tackles the challenge of cross-scale, tri-modal renal biopsy image fusion by introducing CMUS-Net, which combines a Sparse Multi-Instance Learning (SMIL) module to aggregate TEM nanoscale information with a Cross-Modal Scale Attention (CMSA) module to fuse and emphasize cross-modal features from OM and IM. By employing a weighted loss that balances modality contributions, the model achieves superior multi-disease classification of IgAN, MN, and LN on in-house data and demonstrates generalization to MN staging. The approach significantly outperforms state-of-the-art multi-modal or multi-scale methods, validating the importance of explicitly handling cross-scale interactions in renal pathology. These results suggest CMUS-Net as a promising assistive tool for pathologists, potentially accelerating accurate tri-modal glomerular disease diagnosis and staging with ultrastructural context.

Abstract

Constructing a multi-modal automatic classification model based on three types of renal biopsy images can assist pathologists in glomerular multi-disease identification. However, the substantial scale difference between transmission electron microscopy (TEM) image features at the nanoscale and optical microscopy (OM) or immunofluorescence microscopy (IM) images at the microscale poses a challenge for existing multi-modal and multi-scale models in achieving effective feature fusion and improving classification accuracy. To address this issue, we propose a cross-modal ultra-scale learning network (CMUS-Net) for the auxiliary diagnosis of multiple glomerular diseases. CMUS-Net utilizes multiple ultrastructural information to bridge the scale difference between nanometer and micrometer images. Specifically, we introduce a sparse multi-instance learning module to aggregate features from TEM images. Furthermore, we design a cross-modal scale attention module to facilitate feature interaction, enhancing pathological semantic information. Finally, multiple loss functions are combined, allowing the model to weigh the importance among different modalities and achieve precise classification of glomerular diseases. Our method follows the conventional process of renal biopsy pathology diagnosis and, for the first time, performs automatic classification of multiple glomerular diseases including IgA nephropathy (IgAN), membranous nephropathy (MN), and lupus nephritis (LN) based on images from three modalities and two scales. On an in-house dataset, CMUS-Net achieves an ACC of 95.37+/-2.41%, an AUC of 99.05+/-0.53%, and an F1-score of 95.32+/-2.41%. Extensive experiments demonstrate that CMUS-Net outperforms other well-known multi-modal or multi-scale methods and show its generalization capability in staging MN. Code is available at https://github.com/SMU-GL-Group/MultiModal_lkx/tree/main.

Figures (9)

• Figure 1: Illustration of major challenges in classifying glomerular diseases based on renal biopsy images: (a) Subtle pathological changes necessitate TEM images to discern. For example, the deposition locations of immune complexes vary among different diseases, with deposits occurring in the mesangium for IgAN, the glomerular basement membrane (GBM) for MN, and both locations for LN. These details are difficult to distinguish under OM and IM (as indicated by arrows), and precise identification of the deposition locations necessitates TEM. The blue lines in TEM images outline the main areas of these immune complexes deposition, also named electron-dense deposits (EDDs) under TEM. (b) Massive scale difference between modalities. After zooming the same type of detailed lesion areas (such as GBM) in OM and IM images to match those in TEM images, the resolution remains low, and it is still difficult to clearly identify the exact deposition locations of EDDs.
• Figure 2: The overview of the proposed CMUS-Net framework for glomerular disease classification based on multi-modal renal biopsy images.
• Figure 3: Comparison with other attention mechanisms.
• Figure 4: The t-SNE feature clustering results of encoders from different models. The rightmost column shows the clustering results of the fused features. ACC represents the classification accuracy.
• Figure 5: Comparison of our model with the voting-based late fusion model under different batch sizes. The shaded area of the polyline and the error line at the top of the bar represent the standard deviation range of ACC and AUC, respectively.