Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation

Aghiles Kebaili, Jérôme Lapuyade-Lahorgue, Pierre Vera, Su Ruan

TL;DR

The paper addresses missing MR modalities in brain tumor analysis by introducing AMM-Diff, an adaptive diffusion-based imputation framework. It combines a Siamese Image-Frequency Fusion Network (IFFN) trained with masked image modeling to learn a cross-modality representation, with a diffusion decoder that can generate any missing sequences from variable inputs. The approach enables generating multiple missing modalities with a single model by using a fixed-output decoder and an adaptive reconstruction strategy. Evaluations on BRATS 2021 demonstrate improvements over state-of-the-art methods in both quantitative metrics and visual quality, indicating potential to enhance downstream tasks like segmentation. This framework could generalize to other cross-modal imaging modalities and acquisition scenarios, advancing practical clinical imaging.

Abstract

In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs. However, missing MR modalities pose significant challenges for tasks like brain tumor segmentation, especially in deep learning-based segmentation, as each modality provides complementary information crucial for improving accuracy. A promising solution is missing data imputation, where absent modalities are generated from available ones. While generative models have been widely used for this purpose, most state-of-the-art approaches are limited to single or dual target translations, lacking the adaptability to generate missing modalities based on varying input configurations. To address this, we propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones. We designed an Image-Frequency Fusion Network (IFFN) that learns a unified feature representation through a self-supervised pretext task across the full input modalities and their selected high-frequency Fourier components. The proposed diffusion model leverages this representation, encapsulating prior knowledge of the complete modalities, and combines it with an adaptive reconstruction strategy to achieve missing modality completion. Experimental results on the BraTS 2021 dataset demonstrate the effectiveness of our approach.

AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation

TL;DR

The paper addresses missing MR modalities in brain tumor analysis by introducing AMM-Diff, an adaptive diffusion-based imputation framework. It combines a Siamese Image-Frequency Fusion Network (IFFN) trained with masked image modeling to learn a cross-modality representation, with a diffusion decoder that can generate any missing sequences from variable inputs. The approach enables generating multiple missing modalities with a single model by using a fixed-output decoder and an adaptive reconstruction strategy. Evaluations on BRATS 2021 demonstrate improvements over state-of-the-art methods in both quantitative metrics and visual quality, indicating potential to enhance downstream tasks like segmentation. This framework could generalize to other cross-modal imaging modalities and acquisition scenarios, advancing practical clinical imaging.

Abstract

In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs. However, missing MR modalities pose significant challenges for tasks like brain tumor segmentation, especially in deep learning-based segmentation, as each modality provides complementary information crucial for improving accuracy. A promising solution is missing data imputation, where absent modalities are generated from available ones. While generative models have been widely used for this purpose, most state-of-the-art approaches are limited to single or dual target translations, lacking the adaptability to generate missing modalities based on varying input configurations. To address this, we propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones. We designed an Image-Frequency Fusion Network (IFFN) that learns a unified feature representation through a self-supervised pretext task across the full input modalities and their selected high-frequency Fourier components. The proposed diffusion model leverages this representation, encapsulating prior knowledge of the complete modalities, and combines it with an adaptive reconstruction strategy to achieve missing modality completion. Experimental results on the BraTS 2021 dataset demonstrate the effectiveness of our approach.
Paper Structure (10 sections, 3 equations, 3 figures, 1 table)

This paper contains 10 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Diffusion models
  4. Image-Frequency Fusion Network
  5. Adaptive MR sequence completion
  6. Experimentations
  7. Dataset
  8. Quantitative results
  9. Qualitative results
  10. Conclusion

Figures (3)

  • Figure 1: Architecture of the proposed AMM-Diff. (1) Self-supervised pretraining of the Siamese IFFN using a cosine similarity loss. (2) Missing sequence completion. Green-bordered images represent inputs and red-bordered images represent the missing sequences. The input configuration here is entirely random. See section \ref{['sec:IFFN']} and \ref{['sec:diffusion']} for more details.
  • Figure 2: Comparison of translated MR sequences across different methods. Columns represent FLAIR, T1CE, T1, and T2 sequences (left to right).
  • Figure 3: Illustration of adaptive translations from various input configurations. Red-bordered images represent inputs sequences and green-bordered images represent the predicted missing ones. See section \ref{['sec:qualitative']}.