RefineFormer3D: Efficient 3D Medical Image Segmentation via Adaptive Multi-Scale Transformer with Cross Attention Fusion

TL;DR

RefineFormer3D tackles the efficiency-accuracy trade-off in 3D medical image segmentation by introducing a lightweight hierarchical transformer. It combines GhostConv3D-based patch embedding, MixFFN3D with low-rank projections and depthwise 3D convolutions, and an adaptive cross-attention fusion decoder to enable multi-scale feature integration with minimal computational burden. The model achieves competitive Dice scores on BraTS and ACDC with only 2.94M parameters and fast inference (~8.35 ms per volume on GPU), while exhibiting strong robustness to reduced training data and favorable memory usage. These results demonstrate the practicality of transformer-based segmentation in resource-constrained clinical environments and highlight the effectiveness of adaptive skip fusion and efficient feature mixing.

Abstract

Accurate and computationally efficient 3D medical image segmentation remains a critical challenge in clinical workflows. Transformer-based architectures often demonstrate superior global contextual modeling but at the expense of excessive parameter counts and memory demands, restricting their clinical deployment. We propose RefineFormer3D, a lightweight hierarchical transformer architecture that balances segmentation accuracy and computational efficiency for volumetric medical imaging. The architecture integrates three key components: (i) GhostConv3D-based patch embedding for efficient feature extraction with minimal redundancy, (ii) MixFFN3D module with low-rank projections and depthwise convolutions for parameter-efficient feature extraction, and (iii) a cross-attention fusion decoder enabling adaptive multi-scale skip connection integration. RefineFormer3D contains only 2.94M parameters, substantially fewer than contemporary transformer-based methods. Extensive experiments on ACDC and BraTS benchmarks demonstrate that RefineFormer3D achieves 93.44\% and 85.9\% average Dice scores respectively, outperforming or matching state-of-the-art methods while requiring significantly fewer parameters. Furthermore, the model achieves fast inference (8.35 ms per volume on GPU) with low memory requirements, supporting deployment in resource-constrained clinical environments. These results establish RefineFormer3D as an effective and scalable solution for practical 3D medical image segmentation.

Figures (4)

• Figure 1: Model efficiency comparison on ACDC dataset acdc. Parameter count versus segmentation performance for RefineFormer3D and existing 3D segmentation models. Blue bars indicate number of parameters; orange-red curve shows average Dice score. RefineFormer3D achieves superior performance with only 2.94M parameters.
• Figure 2: RefineFormer3D architecture overview. The input volume $\mathbb{R}^{B \times C_{\mathrm{in}} \times D \times H \times W}$ is procecssed via 3D patch embedding and encoded through four hierarchical stages. Each encoder stage applies two consecutive transformer blocks which are windowed self attention followed by shifted window attention. The decoder progressively refines features by fusing encoder skip connections through cross attention, where decoder features query encoder representations for selective multi scale aggregation. The final upsampling block performs refinement without skip connections. Deep supervision is applied via auxiliary heads at intermediate decoder outputs.
• Figure 3: Visual comparison of original MRI, ground truth, and predicted segmentation for four representative BraTS cases showing whole tumor (yellow), enhancing tumor (orange), and tumor core (cyan) regions.
• Figure 4: Visual comparison of original MRI, ground truth, and predicted segmentation for four representative ACDC cardiac cases showing right ventricle (RV), myocardium, and left ventricle (LV).