XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decoder

TL;DR

This paper tackles brain tumor segmentation with incomplete MRI modality data by introducing XLSTM-HVED, which fuses cross-modal information through a Heteromodal Variational Encoder-Decoder and Vision-LSTM attention. Central to the approach are the Self-Attention Variational Encoder (SAVE), the Vision XLSTM Attention Module (ViLA), and the Squeeze-Fusion-Excitation Cross Awareness (SFECA) mechanism, which collectively enable simultaneous reconstruction of missing modalities and accurate segmentation. Experiments on BraTS 2024 demonstrate robust performance under modality absence and strong multimodal fusion capabilities, with ablations confirming the necessity of each component. The proposed method advances practical brain tumor analysis by improving reliability and coherence between reconstruction and segmentation across diverse modality configurations, with code released for reproducibility.

Abstract

Neurogliomas are among the most aggressive forms of cancer, presenting considerable challenges in both treatment and monitoring due to their unpredictable biological behavior. Magnetic resonance imaging (MRI) is currently the preferred method for diagnosing and monitoring gliomas. However, the lack of specific imaging techniques often compromises the accuracy of tumor segmentation during the imaging process. To address this issue, we introduce the XLSTM-HVED model. This model integrates a hetero-modal encoder-decoder framework with the Vision XLSTM module to reconstruct missing MRI modalities. By deeply fusing spatial and temporal features, it enhances tumor segmentation performance. The key innovation of our approach is the Self-Attention Variational Encoder (SAVE) module, which improves the integration of modal features. Additionally, it optimizes the interaction of features between segmentation and reconstruction tasks through the Squeeze-Fusion-Excitation Cross Awareness (SFECA) module. Our experiments using the BraTS 2024 dataset demonstrate that our model significantly outperforms existing advanced methods in handling cases where modalities are missing. Our source code is available at https://github.com/Quanato607/XLSTM-HVED.

Figures (2)

• Figure 1: The overall framework of the proposed XLSTM-HVED model. The model comprises the Self-Attention Variational Encoder, Vision XLSTM Attention Module, and Squeeze-Fusion-Excitation Cross Awareness Module.
• Figure 2: (left) Detailed structure of the Vision-LSTM module. (right) Detailed structure of the cSFE and sSFE modules.