Towards Label-Free Brain Tumor Segmentation: Unsupervised Learning with Multimodal MRI

TL;DR

This work tackles label-free brain tumor segmentation by casting anomaly detection as learning a healthy-prior model from multimodal MRI. It introduces the Multimodal Vision Transformer Autoencoder (MViT-AE) that reconstructs healthy brain MRIs and localizes tumors via reconstruction errors, enhanced by a multimodal early-late fusion strategy and a SAM-based postprocessing stage. The method yields lesion-wise DSCs of ET, NET, SNFH, TC, and WT on BraTS-GoAT 2025, with a 89.4% anomaly Detection Rate on the validation set, demonstrating competitive unsupervised performance and potential scalability across tumor types. Despite challenges such as detecting small or hypointense lesions and the dependence on postprocessing, the approach shows promise for label-efficient neuro-oncological imaging and lays groundwork for future representation learning and semi-supervised refinements.

Abstract

Unsupervised anomaly detection (UAD) presents a complementary alternative to supervised learning for brain tumor segmentation in magnetic resonance imaging (MRI), particularly when annotated datasets are limited, costly, or inconsistent. In this work, we propose a novel Multimodal Vision Transformer Autoencoder (MViT-AE) trained exclusively on healthy brain MRIs to detect and localize tumors via reconstruction-based error maps. This unsupervised paradigm enables segmentation without reliance on manual labels, addressing a key scalability bottleneck in neuroimaging workflows. Our method is evaluated in the BraTS-GoAT 2025 Lighthouse dataset, which includes various types of tumors such as gliomas, meningiomas, and pediatric brain tumors. To enhance performance, we introduce a multimodal early-late fusion strategy that leverages complementary information across multiple MRI sequences, and a post-processing pipeline that integrates the Segment Anything Model (SAM) to refine predicted tumor contours. Despite the known challenges of UAD, particularly in detecting small or non-enhancing lesions, our method achieves clinically meaningful tumor localization, with lesion-wise Dice Similarity Coefficient of 0.437 (Whole Tumor), 0.316 (Tumor Core), and 0.350 (Enhancing Tumor) on the test set, and an anomaly Detection Rate of 89.4% on the validation set. These findings highlight the potential of transformer-based unsupervised models to serve as scalable, label-efficient tools for neuro-oncological imaging.

Figures (5)

• Figure 1: Overview of the proposed Unsupervised Anomaly Segmentation pipeline, comprising preprocessing, transformer-based reconstruction, and SAM-guided postprocessing. Multimodal MRI slices are used to learn healthy anatomical priors, and reconstruction deviations are used to infer tumor regions.
• Figure 2: Overview of the Multimodal ViT-AE (MViT-AE) architecture. The encoder processes multiple input modalities through transformer layers, followed by a fusion layer. The decoder reconstructs each modality, and postprocessing with fusion masking is used to generate the final segmentation.
• Figure 3: Postprocessing pipeline: The original and reconstructed images are subtracted to highlight differences. After thresholding and binarization, small irrelevant regions are removed. A 3D connected component algorithm selects the largest region, which is then refined using the Segment Anything Model (SAM) for improved segmentation accuracy.
• Figure 4: Comparison of segmentation stages using the Segment Anything Model (SAM). (a) Original MRI slice. (b) Ground truth mask showing the expert-annotated tumor. (c) Initial SAM segmentation without prompts, showing poor accuracy. (d) Refined SAM segmentation using bounding box and point prompts, with clearer tumor boundaries.
• Figure 5: Qualitative segmentation results for volume 00018 from the BraTS-MEN 2023 dataset. The first three columns show the original input modalities (T1c, T2f) and the ground truth. The following four columns present the outputs of different model branches: MViT-AE with only T1c, only T2f, both combined, and the combination refined using SAM. Each row corresponds to a different slice from the same volume.