Focal Modulation and Bidirectional Feature Fusion Network for Medical Image Segmentation

TL;DR

The paper tackles the challenge of capturing global context in medical image segmentation by proposing FM-BFF-Net, a hybrid CNN–transformer architecture. It integrates a Focal Modulation–based ConvFormer Attention Block (FMCAB) to refine encoder–decoder feature flows, a Bidirectional Feature Fusion Module (BiFFM) for multi-scale context enrichment, and a Vision Transformer Module (ViTM) at the bottleneck to model long-range dependencies. Powered by EfficientNetV2S1 as the backbone, the approach demonstrates improved boundary precision and robustness across polyp, skin lesion, and ultrasound segmentation tasks, validated on eight public datasets with superior Jaccard and Dice scores compared to several SOTA methods. The proposed framework offers a practical, scalable solution for diverse MIS scenarios and lays groundwork for multilingual, multimodal, and uncertainty-aware clinical deployments.

Abstract

Medical image segmentation is essential for clinical applications such as disease diagnosis, treatment planning, and disease development monitoring because it provides precise morphological and spatial information on anatomical structures that directly influence treatment decisions. Convolutional neural networks significantly impact image segmentation; however, since convolution operations are local, capturing global contextual information and long-range dependencies is still challenging. Their capacity to precisely segment structures with complicated borders and a variety of sizes is impacted by this restriction. Since transformers use self-attention methods to capture global context and long-range dependencies efficiently, integrating transformer-based architecture with CNNs is a feasible approach to overcoming these challenges. To address these challenges, we propose the Focal Modulation and Bidirectional Feature Fusion Network for Medical Image Segmentation, referred to as FM-BFF-Net in the remainder of this paper. The network combines convolutional and transformer components, employs a focal modulation attention mechanism to refine context awareness, and introduces a bidirectional feature fusion module that enables efficient interaction between encoder and decoder representations across scales. Through this design, FM-BFF-Net enhances boundary precision and robustness to variations in lesion size, shape, and contrast. Extensive experiments on eight publicly available datasets, including polyp detection, skin lesion segmentation, and ultrasound imaging, show that FM-BFF-Net consistently surpasses recent state-of-the-art methods in Jaccard index and Dice coefficient, confirming its effectiveness and adaptability for diverse medical imaging scenarios.

Figures (12)

• Figure 1: Overview of the proposed M-BFF-Net architecture for medical image segmentation. The model integrates convolutional and transformer-based modules to capture both local and global features. It consists of four encoder blocks, each followed by a Focal Modulation-based ConvFormer Attention Block (FMCAB). The encoded features are refined through a Vision Transformer and progressively decoded using Feature Refinement Modules (FRMs) and Bidirectional Feature Fusion Modules (BiFFMs). Skip connections and multi-scale fusion enhance the contextual representation and boundary precision of segmentation outputs.
• Figure 2: (a) Architecture of the proposed Focal Modulation-based ConvFormer Attention Block (FMCAB), which combines convolutional and attention mechanisms to enhance spatial and contextual feature representation. The block leverages global average pooling (GAP), reshaping operations, and focal modulation to adaptively refine features. (b) Internal structure of the Focal Modulation (FM) unit, which computes attention weights by fusing outputs from global average pooling (GAP) and global max pooling (GMP), followed by nonlinear transformations for precise focus modulation.
• Figure 3: Detailed schematic of the proposed Bidirectional Feature Fusion Module (BiFFM). This module integrates multi-scale features from encoder and decoder branches using global average pooling (GAP) and parallel convolutional pathways. Features $X_1$ and $X_2$ are first aggregated and then refined through channel-wise operations and channel shuffle to enhance inter-feature interactions. The fusion output $f_{fuse}$ is obtained by concatenating and modulating these feature maps, enabling effective bidirectional information exchange and improved semantic representation.
• Figure 4: Architectural schematic of the proposed Vision Transformer Module (ViTM). The module consists of two main components: the Global Self-Attention (GSA) mechanism and the Token-based Self-Attention (TSA) module. GSA extracts global contextual relationships through multi-branch attention and hierarchical feature refinement. TSA operates on tokenized representations with positional encoding, utilizing multi-head self-attention via query ($Q$), key ($K$), and value ($V$) embeddings. The final refined feature map $F_{\text{out}}$ is generated through concatenation and permutation operations, allowing for rich global feature learning in medical image segmentation.
• Figure 5: Visual performance comparison of the proposed M-BFF-Net on Kvasir-SEG dataset.