Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large-scale modality-invariant foundation models for brain MRI analysis: Application to lesion segmentation

Petros Koutsouvelis, Matej Gazda, Leroy Volmer, Sina Amirrajab, Kamil Barbierik, Branislav Setlak, Jakub Gazda, Peter Drotar

TL;DR

This work examines large-scale self-supervised pre-training for brain MRI using a modality-invariant representation objective (MCL) integrated with masked image modeling (MIM) to support lesion segmentation in stroke and epilepsy. While MCL achieves cross-modality embedding alignment, it does not improve per-modality lesion segmentation beyond a strong MIM baseline; segmentation performance remains highly modality-dependent, underscoring the importance of modality-specific texture and contrast. The study demonstrates that invariant representations may excel for global brain tasks rather than nuanced local pathology, and provides pretrained checkpoints to foster future research. Overall, the findings guide the design of neuroimaging foundation models toward balancing cross-modality alignment with preservation of modality-specific information for segmentation tasks.

Abstract

The field of computer vision is undergoing a paradigm shift toward large-scale foundation model pre-training via self-supervised learning (SSL). Leveraging large volumes of unlabeled brain MRI data, such models can learn anatomical priors that improve few-shot performance in diverse neuroimaging tasks. However, most SSL frameworks are tailored to natural images, and their adaptation to capture multi-modal MRI information remains underexplored. This work proposes a modality-invariant representation learning setup and evaluates its effectiveness in stroke and epilepsy lesion segmentation, following large-scale pre-training. Experimental results suggest that despite successful cross-modality alignment, lesion segmentation primarily benefits from preserving fine-grained modality-specific features. Model checkpoints and code are made publicly available.

Large-scale modality-invariant foundation models for brain MRI analysis: Application to lesion segmentation

TL;DR

This work examines large-scale self-supervised pre-training for brain MRI using a modality-invariant representation objective (MCL) integrated with masked image modeling (MIM) to support lesion segmentation in stroke and epilepsy. While MCL achieves cross-modality embedding alignment, it does not improve per-modality lesion segmentation beyond a strong MIM baseline; segmentation performance remains highly modality-dependent, underscoring the importance of modality-specific texture and contrast. The study demonstrates that invariant representations may excel for global brain tasks rather than nuanced local pathology, and provides pretrained checkpoints to foster future research. Overall, the findings guide the design of neuroimaging foundation models toward balancing cross-modality alignment with preservation of modality-specific information for segmentation tasks.

Abstract

The field of computer vision is undergoing a paradigm shift toward large-scale foundation model pre-training via self-supervised learning (SSL). Leveraging large volumes of unlabeled brain MRI data, such models can learn anatomical priors that improve few-shot performance in diverse neuroimaging tasks. However, most SSL frameworks are tailored to natural images, and their adaptation to capture multi-modal MRI information remains underexplored. This work proposes a modality-invariant representation learning setup and evaluates its effectiveness in stroke and epilepsy lesion segmentation, following large-scale pre-training. Experimental results suggest that despite successful cross-modality alignment, lesion segmentation primarily benefits from preserving fine-grained modality-specific features. Model checkpoints and code are made publicly available.

Paper Structure

This paper contains 13 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Data Curation and Pre-processing
  4. Modality-Invariant Contrastive Learning (MCL)
  5. Integration with Masked Image Modeling (MIM)
  6. Implementation Details
  7. Evaluation Setup
  8. Experimental Results
  9. Modality Invariance in the Embedding Space
  10. Lesion Segmentation Performance
  11. Modality Invariance in Downstream Performance
  12. Conclusion
  13. Acknowledgments

Figures (3)

  • Figure 1: High-level schematic of the pre-training setup. Abbreviations: EMA = exponential moving average.
  • Figure 2: tSNE on the embeddings from 30 randomly selected cases of the FOMO60k validation set using the MCL and CL models.
  • Figure 3: Segmentation DSC in the BOHN FCD and ISLES 2022 datasets for each modality and pre-training method.