SSPFormer: Self-Supervised Pretrained Transformer for MRI Images

TL;DR

This work tackles the domain adaptation, data scarcity, and artifact robustness challenges of applying large pretrained transformers to MRI. It introduces SSPFormer, a self-supervised Transformer pretrained on unlabeled MRI data with three frequency-aware objectives—inverse frequency masking, frequency-weighted FFT noise augmentation, and cross-modality frequency attention—on a large multi-organ MRI corpus, using a frozen encoder and a lightweight decoder with about $0.9$M trainable parameters. The authors validate SSPFormer across segmentation, super-resolution, and denoising, showing state-of-the-art performance with only $20 ext%$ labeled data and strong robustness to real MRI artifacts, supported by ablation and a radiologist user study. This approach yields a data-efficient, privacy-conscious MRI backbone that can be rapidly deployed for multiple clinical tasks with minimal annotation effort.

Abstract

The pre-trained transformer demonstrates remarkable generalization ability in natural image processing. However, directly transferring it to magnetic resonance images faces two key challenges: the inability to adapt to the specificity of medical anatomical structures and the limitations brought about by the privacy and scarcity of medical data. To address these issues, this paper proposes a Self-Supervised Pretrained Transformer (SSPFormer) for MRI images, which effectively learns domain-specific feature representations of medical images by leveraging unlabeled raw imaging data. To tackle the domain gap and data scarcity, we introduce inverse frequency projection masking, which prioritizes the reconstruction of high-frequency anatomical regions to enforce structure-aware representation learning. Simultaneously, to enhance robustness against real-world MRI artifacts, we employ frequency-weighted FFT noise enhancement that injects physiologically realistic noise into the Fourier domain. Together, these strategies enable the model to learn domain-invariant and artifact-robust features directly from raw scans. Through extensive experiments on segmentation, super-resolution, and denoising tasks, the proposed SSPFormer achieves state-of-the-art performance, fully verifying its ability to capture fine-grained MRI image fidelity and adapt to clinical application requirements.

Figures (8)

• Figure 1: Schematic diagram of the proposed self-supervised pretrained Transformer framework tailored for MRI images. The framework integrates core components, including frequency-aware hierarchical masking and Fourier domain noise augmentation, aiming to learn domain-specific feature representations of MRI images efficiently.
• Figure 2: Pipeline of the proposed self-supervised pretrained Transformer framework for MRI images (SSPFormer). The framework integrates core components, including inverse frequency-aware masking and frequency-weighted Fourier noise augmentation, enabling customized learning of domain-specific feature representations for MRI images.
• Figure 3: Illustration of the asymmetric fine-tuning strategy: freezing the encoder and fine-tuning only the decoder and task-specific heads. This pre-trained Transformer can accurately perform a variety of visual tasks on MRI images.
• Figure 4: Sample MRI images from our multi-source dataset, including glioma, meningioma, pituitary tumor, and normal brain cases, illustrating the heterogeneity of anatomical and pathological variations in clinical neuroimaging.
• Figure 5: The performance of the brain tumor segmentation task.