Phi-SegNet: Phase-Integrated Supervision for Medical Image Segmentation

TL;DR

This work addresses the challenge of robust cross-modality semantic segmentation in medical images by introducing Phi-SegNet, a CNN-based framework that integrates phase-aware supervision with spectral priors. The architecture combines Bi-Feature Mask Former modules for multi-scale feature fusion, Reverse Fourier Attention blocks for frequency-domain refinement, and a phase-integrated loss to enforce boundary coherence in the Fourier domain. Across five diverse datasets, Phi-SegNet delivers state-of-the-art IoU and F1 scores, with strong boundary fidelity and competitive cross-dataset generalization, highlighting the practical benefits of jointly leveraging spatial and phase information. The findings suggest that phase-conditioned propagation and spectral supervision can significantly enhance fine-grained object localization in MIS and pave the way for more generalizable, spectrum-aware segmentation models.

Abstract

Deep learning has substantially advanced medical image segmentation, yet achieving robust generalization across diverse imaging modalities and anatomical structures remains a major challenge. A key contributor to this limitation lies in how existing architectures, ranging from CNNs to Transformers and their hybrids, primarily encode spatial information while overlooking frequency-domain representations that capture rich structural and textural cues. Although few recent studies have begun exploring spectral information at the feature level, supervision-level integration of frequency cues-crucial for fine-grained object localization-remains largely untapped. To this end, we propose Phi-SegNet, a CNN-based architecture that incorporates phase-aware information at both architectural and optimization levels. The network integrates Bi-Feature Mask Former (BFMF) modules that blend neighboring encoder features to reduce semantic gaps, and Reverse Fourier Attention (RFA) blocks that refine decoder outputs using phase-regularized features. A dedicated phase-aware loss aligns these features with structural priors, forming a closed feedback loop that emphasizes boundary precision. Evaluated on five public datasets spanning X-ray, US, histopathology, MRI, and colonoscopy, Phi-SegNet consistently achieved state-of-the-art performance, with an average relative improvement of 1.54+/-1.26% in IoU and 0.98+/-0.71% in F1-score over the next best-performing model. In cross-dataset generalization scenarios involving unseen datasets from the known domain, Phi-SegNet also exhibits robust and superior performance, highlighting its adaptability and modality-agnostic design. These findings demonstrate the potential of leveraging spectral priors in both feature representation and supervision, paving the way for generalized segmentation frameworks that excel in fine-grained object localization.

Figures (7)

• Figure 1: Overview of the proposed Phi-SegNet. The architecture integrates encoder features via bi-feature mask former (BFMF) modules and attention-guided skip connections. Phase supervision is applied on decoder stages followed by reverse Fourier attention (R$\mathcal{F}$A) modules, which use spectral filtering to enhance boundary localization.
• Figure 2: The Bi-Feature Mask Former (BFMF) module extracts multi-scale semantic and spatial features through multi-kernel convolutions (1×1, 3×3, 5×5) and sequential aggregation.
• Figure 3: Attention-guided fusion strategy. At each encoder stage $E_i$, the attention mask $y^{E_i}$ is combined with the max-pooled mask from the preceding stage $y^{E_{i-1}}_s$, and passed through a convolutional block followed by a sigmoid activation to generate a refined attention map. This map modulates the encoder features via element-wise multiplication and is subsequently added to the decoder pathway.
• Figure 4: The Reverse Fourier Attention (R$\mathcal{F}$A) module applies a $\gamma^2$-weighted low-pass filter in the frequency domain to enforce phase-conditioned boundary enhancement. Reverse attention focuses on what the model misses and helps it refine low-confidence regions like boundaries or tiny lesions. Early decoder features are coarse, applying LPF in Fourier space suppresses high-frequency noise, allowing attention to act on structurally dominant, globally coherent features.
• Figure 5: Visualization of segmentation results obtained by Phi-SegNet and other state-of-the-art architectures (two test cases from each dataset). Certain regions in some of the images have been marked using blue bounding boxes to highlight the differences for the convenience of the reader.