Decoding with Structured Awareness: Integrating Directional, Frequency-Spatial, and Structural Attention for Medical Image Segmentation

TL;DR

The paper tackles the challenge of preserving fine-grained edges and global structure in medical image segmentation by introducing a three-module decoder: ACFA for directional, edge-aware attention; TFFA for joint spatial-frequency representation via spatial, Fourier, and wavelet branches; and SMMM for structure-aware, multi-scale skip fusion. The approach achieves superior segmentation accuracy across diverse tasks (skin lesions, abdominal organs, cardiac structures) and datasets (ISIC 2017/2018, Synapse, ACDC), with ablations confirming the contribution and synergy of each module. Quantitative results show state-of-the-art or competitive DSC improvements, complemented by qualitative attention heatmaps illustrating enhanced boundary delineation. Overall, the work delivers a practical, efficient decoder design that enhances high-precision medical segmentation and demonstrates strong generalization across multimodal biomedical imaging tasks.

Abstract

To address the limitations of Transformer decoders in capturing edge details, recognizing local textures and modeling spatial continuity, this paper proposes a novel decoder framework specifically designed for medical image segmentation, comprising three core modules. First, the Adaptive Cross-Fusion Attention (ACFA) module integrates channel feature enhancement with spatial attention mechanisms and introduces learnable guidance in three directions (planar, horizontal, and vertical) to enhance responsiveness to key regions and structural orientations. Second, the Triple Feature Fusion Attention (TFFA) module fuses features from Spatial, Fourier and Wavelet domains, achieving joint frequency-spatial representation that strengthens global dependency and structural modeling while preserving local information such as edges and textures, making it particularly effective in complex and blurred boundary scenarios. Finally, the Structural-aware Multi-scale Masking Module (SMMM) optimizes the skip connections between encoder and decoder by leveraging multi-scale context and structural saliency filtering, effectively reducing feature redundancy and improving semantic interaction quality. Working synergistically, these modules not only address the shortcomings of traditional decoders but also significantly enhance performance in high-precision tasks such as tumor segmentation and organ boundary extraction, improving both segmentation accuracy and model generalization. Experimental results demonstrate that this framework provides an efficient and practical solution for medical image segmentation.

Figures (4)

• Figure 1: Proposed network architecture of the model.
• Figure 2: Comparison of the performance of our model with the visualization results of other models on the Synapse dataset.
• Figure 3: Comparison of segmentation results on the ISIC 2017 dataset with other previous methods.
• Figure 4: Attention heatmap visualization for the proposed model on ISIC 2018.