Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

multiGradICON: A Foundation Model for Multimodal Medical Image Registration

Basar Demir, Lin Tian, Thomas Hastings Greer, Roland Kwitt, Francois-Xavier Vialard, Raul San Jose Estepar, Sylvain Bouix, Richard Jarrett Rushmore, Ebrahim Ebrahim, Marc Niethammer

TL;DR

multiGradICON tackles universal multimodal medical image registration by extending the uniGradICON framework with a multimodal similarity measure and training-time loss randomization across modalities. It adopts the GradICON architectural paradigm with a multi-resolution TS/DS pipeline and trains on a large, diverse corpus spanning 16 datasets, 5 anatomical regions, and 12 modalities, using a $1-\text{LNCC}^2$ loss to handle sign-ambiguous multimodal correlations. The results show that multiGradICON retains strong monomodal performance while achieving pronounced improvements in multimodal registration, particularly when employing loss-randomization strategies, with ROI-focused training further boosting lung registrations; some challenging cross-modality pairs (e.g., DIXON fat–water) remain difficult and may benefit from segmentation-based losses. The work positions multiGradICON as a foundation-model-style approach to broad cross-modality alignment in medical imaging, enabling broader generalization across anatomies and imaging modalities, and it provides public code to foster accessibility and further research.

Abstract

Modern medical image registration approaches predict deformations using deep networks. These approaches achieve state-of-the-art (SOTA) registration accuracy and are generally fast. However, deep learning (DL) approaches are, in contrast to conventional non-deep-learning-based approaches, anatomy-specific. Recently, a universal deep registration approach, uniGradICON, has been proposed. However, uniGradICON focuses on monomodal image registration. In this work, we therefore develop multiGradICON as a first step towards universal *multimodal* medical image registration. Specifically, we show that 1) we can train a DL registration model that is suitable for monomodal *and* multimodal registration; 2) loss function randomization can increase multimodal registration accuracy; and 3) training a model with multimodal data helps multimodal generalization. Our code and the multiGradICON model are available at https://github.com/uncbiag/uniGradICON.

multiGradICON: A Foundation Model for Multimodal Medical Image Registration

TL;DR

multiGradICON tackles universal multimodal medical image registration by extending the uniGradICON framework with a multimodal similarity measure and training-time loss randomization across modalities. It adopts the GradICON architectural paradigm with a multi-resolution TS/DS pipeline and trains on a large, diverse corpus spanning 16 datasets, 5 anatomical regions, and 12 modalities, using a loss to handle sign-ambiguous multimodal correlations. The results show that multiGradICON retains strong monomodal performance while achieving pronounced improvements in multimodal registration, particularly when employing loss-randomization strategies, with ROI-focused training further boosting lung registrations; some challenging cross-modality pairs (e.g., DIXON fat–water) remain difficult and may benefit from segmentation-based losses. The work positions multiGradICON as a foundation-model-style approach to broad cross-modality alignment in medical imaging, enabling broader generalization across anatomies and imaging modalities, and it provides public code to foster accessibility and further research.

Abstract

Modern medical image registration approaches predict deformations using deep networks. These approaches achieve state-of-the-art (SOTA) registration accuracy and are generally fast. However, deep learning (DL) approaches are, in contrast to conventional non-deep-learning-based approaches, anatomy-specific. Recently, a universal deep registration approach, uniGradICON, has been proposed. However, uniGradICON focuses on monomodal image registration. In this work, we therefore develop multiGradICON as a first step towards universal *multimodal* medical image registration. Specifically, we show that 1) we can train a DL registration model that is suitable for monomodal *and* multimodal registration; 2) loss function randomization can increase multimodal registration accuracy; and 3) training a model with multimodal data helps multimodal generalization. Our code and the multiGradICON model are available at https://github.com/uncbiag/uniGradICON.
Paper Structure (13 sections, 3 equations, 2 figures, 9 tables)

This paper contains 13 sections, 3 equations, 2 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Architecture
  5. Training losses
  6. Dataset
  7. Results
  8. Performance on monomodal registration
  9. Performance on multimodal registration
  10. Conclusion
  11. Acknowledgements
  12. Additional visualizations
  13. Erratum

Figures (2)

  • Figure 1: Comparison of uni- and multiGradICON on T1w MRI-mean diffusivity (MD) registration from ABCD. Note the much better matching of the ventricles for multiGradICON.
  • Figure 2: Visualizations of monomodal and multimodal registration results for multiGradICON without instance optimization. The results demonstrate that multiGradICON can handle a wide variety of modalities and anatomies with smooth displacement fields. Note that these images are visualized in their interpolated shapes as they are provided to the network. DIXON images reproduced by kind permission of the UK Biobank®.