An Arbitrary-Modal Fusion Network for Volumetric Cranial Nerves Tract Segmentation

TL;DR

CNTSeg-v2 tackles cranial nerves tract segmentation under incomplete multimodal MRI data by using T1w as the primary modality to guide auxiliary modalities through an Arbitrary-Modal Collaboration Module (ACM). It couples this with a Deep Distance-guided Multi-stage (DDM) decoder that employs Signed Distance Map supervision to refine boundaries. The approach reports state-of-the-art performance on the HCP and MDM datasets and demonstrates robustness to missing modalities, achieving superior segmentation metrics and qualitative fidelity. This framework facilitates practical CN tract segmentation in real-world clinical settings, enabling more reliable preoperative planning with partially available multimodal data. The learning objective combines segmentation and shape terms via ${\mathcal{L}}_{total} = \lambda {\mathcal{L}}_{seg} + (1-\lambda) {\mathcal{L}}_{shape}$, promoting accurate interior labeling and precise boundaries.

Abstract

The segmentation of cranial nerves (CNs) tract provides a valuable quantitative tool for the analysis of the morphology and trajectory of individual CNs. Multimodal CNs tract segmentation networks, e.g., CNTSeg, which combine structural Magnetic Resonance Imaging (MRI) and diffusion MRI, have achieved promising segmentation performance. However, it is laborious or even infeasible to collect complete multimodal data in clinical practice due to limitations in equipment, user privacy, and working conditions. In this work, we propose a novel arbitrary-modal fusion network for volumetric CNs tract segmentation, called CNTSeg-v2, which trains one model to handle different combinations of available modalities. Instead of directly combining all the modalities, we select T1-weighted (T1w) images as the primary modality due to its simplicity in data acquisition and contribution most to the results, which supervises the information selection of other auxiliary modalities. Our model encompasses an Arbitrary-Modal Collaboration Module (ACM) designed to effectively extract informative features from other auxiliary modalities, guided by the supervision of T1w images. Meanwhile, we construct a Deep Distance-guided Multi-stage (DDM) decoder to correct small errors and discontinuities through signed distance maps to improve segmentation accuracy. We evaluate our CNTSeg-v2 on the Human Connectome Project (HCP) dataset and the clinical Multi-shell Diffusion MRI (MDM) dataset. Extensive experimental results show that our CNTSeg-v2 achieves state-of-the-art segmentation performance, outperforming all competing methods.

Figures (11)

• Figure 1: Comparing different single modalities (i.e., T1w images, T2w images, FA images, DEC images, and Peaks images), multimodal fusion networks including MMFnet xie2023deep, CNTSeg xie2023cntseg, and our CNTSeg-v2 on the HCP dataset.
• Figure 2: Overview of our CNTSeg-v2. The encoder is two-branch, which correspond to the primary branch for T1w images and the secondary branch for other other modalities (i.e., T2w, FA, Peaks, and DEC). The decoder is divided into two branches: Deep Distance-guided Multi-stage (DDM) branch and semantic supervision branch. The ACM selects informative supplementary features from other auxiliary available modalities. The input T1w feature ${\mathbf{F}_i}^{T1}$ and each complementary feature $\left\{{\mathbf{F}_i}^{M}, M \in \left\{ {\mathrm{T2},\mathrm{FA},\mathrm{Peaks},\mathrm{DEC}} \right\}\right\}$ are sent to the information scoring module to get the fused features ${\mathbf{\hat{F}}_i}^{T1}$. The Dual-Attention Feature-Interactive Module (DFM) leverages both spatial and channel attention to aggregate multi-modal features.
• Figure 3: The overall of the Deep Distance-guided Multi-stage (DDM) decoder.
• Figure 4: Qualitative results comparison of SOTA methods on HCP dataset. In each subfigure, the 1st and 3rd show two different slices of the segmentation results for each method in one subject, while the 2nd and 4th are the corresponding enlarged images. Green shows the reference data and red shows the segmentation of the respective method.
• Figure 5: Box-plots of segmentation results comparing our CNTSeg-v2 with SOTA cranial nerves segmentation methods on MDM dataset.