SYNAPSE-Net: A Unified Framework with Lesion-Aware Hierarchical Gating for Robust Segmentation of Heterogeneous Brain Lesions

TL;DR

SYNAPSE-Net proposes a unified multi-stream CNN-Transformer framework for robust segmentation of heterogeneous brain lesions across multi-modal MRI. It combines parallel modality encoders, a Swin Transformer-based global context bottleneck, adaptive cross-modal attention fusion, and a hierarchical gated UNet++ decoder to produce high-fidelity lesion masks. The approach is supported by variance-reduction training with difficulty-aware sampling and pathology-specific loss configurations, and is validated on WMH, ISLES, and BraTS benchmarks where it achieves state-of-the-art or competitive performance with strong boundary accuracy and low variance. The findings suggest that a single, generalizable architecture can outperform specialized models across diverse brain pathologies, holding promise for scalable clinical deployment and cross-task reliability.

Abstract

Automated segmentation of heterogeneous brain lesions from multi-modal MRI remains a critical challenge in clinical neuroimaging. Current deep learning models are typically specialized `point solutions' that lack generalization and high performance variance, limiting their clinical reliability. To address these gaps, we propose the Unified Multi-Stream SYNAPSE-Net, an adaptive framework designed for both generalization and robustness. The framework is built on a novel hybrid architecture integrating multi-stream CNN encoders, a Swin Transformer bottleneck for global context, a dynamic cross-modal attention fusion (CMAF) mechanism, and a hierarchical gated decoder for high-fidelity mask reconstruction. The architecture is trained with a variance reduction strategy that combines pathology specific data augmentation and difficulty-aware sampling method. The model was evaluated on three different challenging public datasets: the MICCAI 2017 WMH Challenge, the ISLES 2022 Challenge, and the BraTS 2020 Challenge. Our framework attained a state-of-the-art DSC value of 0.831 with the HD95 value of 3.03 in the WMH dataset. For ISLES 2022, it achieved the best boundary accuracy with a statistically significant difference (HD95 value of 9.69). For BraTS 2020, it reached the highest DSC value for the tumor core region (0.8651). These experimental findings suggest that our unified adaptive framework achieves state-of-the-art performance across multiple brain pathologies, providing a robust and clinically feasible solution for automated segmentation. The source code and the pre-trained models are available at https://github.com/mubid-01/SYNAPSE-Net-pre.

Figures (9)

• Figure 1: The SYNAPSE-Net architecture, consisting of N parallel CNN encoders, a hybrid bottleneck that fuses modalities using Swin Transformers and Cross-Modal Attention, and a Hierarchical Gated Decoder with a UNet++ backbone to generate the final segmentation.
• Figure 2: The five-stage Encoder for extracting five levels of feature maps ($f_1$ to $f_5$) for each input modality.
• Figure 3: Conceptual diagram of the Hierarchical Gated Decoder, where Lesion Gate modules refine skip connections ($f_1$–$f_4$) before feeding them into the dense UNet++ decoder nodes ($X_{i,j}$).
• Figure 4: The hierarchical gated decoder realization using the dense UNet++ backbone, where the gated skip connections ($f_{i,\text{gated}}$) and bottleneck tensor are utilized.
• Figure 5: Qualitative results of the architectural ablation study on the WMH dataset. Segmentation overlays show true positives (green), false negatives (red), and false positives (blue).