DAMM-Diffusion: Learning Divergence-Aware Multi-Modal Diffusion Model for Nanoparticles Distribution Prediction

TL;DR

The experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods, and the proposed Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map.

Abstract

The prediction of nanoparticles (NPs) distribution is crucial for the diagnosis and treatment of tumors. Recent studies indicate that the heterogeneity of tumor microenvironment (TME) highly affects the distribution of NPs across tumors. Hence, it has become a research hotspot to generate the NPs distribution by the aid of multi-modal TME components. However, the distribution divergence among multi-modal TME components may cause side effects i.e., the best uni-modal model may outperform the joint generative model. To address the above issues, we propose a \textbf{D}ivergence-\textbf{A}ware \textbf{M}ulti-\textbf{M}odal \textbf{Diffusion} model (i.e., \textbf{DAMM-Diffusion}) to adaptively generate the prediction results from uni-modal and multi-modal branches in a unified network. In detail, the uni-modal branch is composed of the U-Net architecture while the multi-modal branch extends it by introducing two novel fusion modules i.e., Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM). Specifically, the MMFM is proposed to fuse features from multiple modalities, while the UAFM module is introduced to learn the uncertainty map for cross-attention computation. Following the individual prediction results from each branch, the Divergence-Aware Multi-Modal Predictor (DAMMP) module is proposed to assess the consistency of multi-modal data with the uncertainty map, which determines whether the final prediction results come from multi-modal or uni-modal predictions. We predict the NPs distribution given the TME components of tumor vessels and cell nuclei, and the experimental results show that DAMM-Diffusion can generate the distribution of NPs with higher accuracy than the comparing methods. Additional results on the multi-modal brain image synthesis task further validate the effectiveness of the proposed method.

Figures (12)

• Figure 1: The illustration of different NPs distribution prediction methods. (a) The Uni-modal method predicts the distribution of NPs by vessels. (b) The multi-modal method predicts the distribution of NPs by the combination of vessels and nuclei. (c) Our DAMM-Diffusion considers both uni-modal and multi-modal branches for NPs distribution prediction by considering the divergence among nuclei and vessels channels.
• Figure 2: Overview of DAMM-Diffusion. At each step of the reverse process, both the uni-modal branch and multi-modal branch perform the reverse step in a unified network, and output the predictions respectively. The uni-modal branch consists of a U-Net architecture with the encoder, middle U-Net bottleneck and a decoder. The multi-modal branch additionally incorporates two modules i.e., MMFM and UAFM aiming at fusing the multi-modal data. Finally, DAMMP decides whether to apply multi-modal predictions or only using the uni-modal generation results for NPs distribution prediction.
• Figure 3: Illustration of the proposed Multi-Modal Fusion Module (MMFM) and Uncertainty-Aware Fusion Module (UAFM).
• Figure 4: Qualitative comparison between the proposed method and the previous methods for NPs distribution at whole-slide level.
• Figure 5: Qualitative comparison between the proposed method and other state-of-the-art methods at patch-level.