S2M-Net: Spectral-Spatial Mixing for Medical Image Segmentation with Morphology-Aware Adaptive Loss

TL;DR

Medical image segmentation must balance boundary precision, global anatomical coherence, and efficient learning on limited clinical data. The authors introduce S2M-Net, a compact architecture that integrates Spectral-Selective Token Mixer for global context with a morphology-aware adaptive loss and a boundary-focused decoder to achieve dense, boundary-accurate predictions at sub-quadratic complexity. MASL automatically tunes five loss components based on per-sample geometry, while SSTM uses aggressive frequency truncation (e.g., $K=32$) to retain energy with much lower computation. Across 16 datasets spanning 8 modalities, S2M-Net delivers state-of-the-art Dice scores (e.g., 96.12% on ETIS-PolypDB, 83.77% on EndoVis17, 80.90% on BRATS) with 3.5–6× fewer parameters than transformer-based methods, demonstrating strong generalization and practical deployment potential.

Abstract

Medical image segmentation requires balancing local precision for boundary-critical clinical applications, global context for anatomical coherence, and computational efficiency for deployment on limited data and hardware a trilemma that existing architectures fail to resolve. Although convolutional networks provide local precision at $\mathcal{O}(n)$ cost but limited receptive fields, vision transformers achieve global context through $\mathcal{O}(n^2)$ self-attention at prohibitive computational expense, causing overfitting on small clinical datasets. We propose S2M-Net, a 4.7M-parameter architecture that achieves $\mathcal{O}(HW \log HW)$ global context through two synergistic innovations: (i) Spectral-Selective Token Mixer (SSTM), which exploits the spectral concentration of medical images via truncated 2D FFT with learnable frequency filtering and content-gated spatial projection, avoiding quadratic attention cost while maintaining global receptive fields; and (ii) Morphology-Aware Adaptive Segmentation Loss (MASL), which automatically analyzes structure characteristics (compactness, tubularity, irregularity, scale) to modulate five complementary loss components through constrained learnable weights, eliminating manual per-dataset tuning. Comprehensive evaluation in 16 medical imaging datasets that span 8 modalities demonstrates state-of-the-art performance: 96.12\% Dice on polyp segmentation, 83.77\% on surgical instruments (+17.85\% over the prior art) and 80.90\% on brain tumors, with consistent 3-18\% improvements over specialized baselines while using 3.5--6$\times$ fewer parameters than transformer-based methods.

Figures (2)

• Figure 1: Comparison of feature interaction mechanisms in medical image segmentation. Left: CNN-based U-Net performs local $K \times K$ aggregation with limited global context. Middle: Transformers apply global self-attention at $\mathcal{O}(n^2)$ cost. Right: S2M-Net achieves efficient global context via spectral-selective token mixing (SSTM) while preserving sharp boundaries through boundary-focused decoding.
• Figure 2: S2M-Net overview. The encoder has five stages with MRF-SE blocks and stage-wise global mixing via SSTM, while the decoder uses a boundary-focused pathway with soft spatial routing to refine region and boundary predictions.