IMPACT: A Generic Semantic Loss for Multimodal Medical Image Registration

TL;DR

IMPACT targets robust multimodal medical image registration by replacing intensity-based similarity with a semantic feature-based metric derived from pretrained segmentation models. It uses a differentiable loss built on features from TotalSegmentator, SAM, and related models, applicable to both algorithmic (Elastix) and DL-based (VoxelMorph) registration without task-specific training. The method introduces two operational modes, Jacobian and Static, enabling efficient optimization or precomputed feature usage, and supports weak supervision with masks and multi-resolution schemes. Across five challenging thoracic, abdominal, and pelvic registration tasks, IMPACT consistently improves TRE, DSC, and HD95 over traditional metrics and several deep learning baselines, demonstrating robustness to noise, artifacts, and modality gaps. The work provides a flexible, plug-and-play framework with public code for broad adoption in clinical and research contexts.

Abstract

Image registration is fundamental in medical imaging, enabling precise alignment of anatomical structures for diagnosis, treatment planning, image-guided interventions, and longitudinal monitoring. This work introduces IMPACT (Image Metric with Pretrained model-Agnostic Comparison for Transmodality registration), a novel similarity metric designed for robust multimodal image registration. Rather than relying on raw intensities, handcrafted descriptors, or task-specific training, IMPACT defines a semantic similarity measure based on the comparison of deep features extracted from large-scale pretrained segmentation models. By leveraging representations from models such as TotalSegmentator, Segment Anything (SAM), and other foundation networks, IMPACT provides a task-agnostic, training-free solution that generalizes across imaging modalities. These features, originally trained for segmentation, offer strong spatial correspondence and semantic alignment capabilities, making them naturally suited for registration. The method integrates seamlessly into both algorithmic (Elastix) and learning-based (VoxelMorph) frameworks, leveraging the strengths of each. IMPACT was evaluated on five challenging 3D registration tasks involving thoracic CT/CBCT and pelvic MR/CT datasets. Quantitative metrics, including Target Registration Error and Dice Similarity Coefficient, demonstrated consistent improvements in anatomical alignment over baseline methods. Qualitative analyses further highlighted the robustness of the proposed metric in the presence of noise, artifacts, and modality variations. With its versatility, efficiency, and strong performance across diverse tasks, IMPACT offers a powerful solution for advancing multimodal image registration in both clinical and research settings.

Figures (6)

• Figure 1: Schematic overview of the proposed feature-based similarity metric the algorithmic registration framework in the 3D case. Features are extracted from the fixed and moving images using a shared pretrained encoder $\phi$. The similarity is computed between features of the fixed image $\phi(I_F(x_i))$ and the transformed moving image $\phi(I_M(T_\theta(x_i)))$ using a distance function $\mathcal{D}$. Two modes are supported: Jacobian Mode, where local patches are sampled and gradients are backpropagated through the feature extractor, and Static Mode, where features are precomputed over the entire image and reused throughout the optimization. Gradients are used to iteratively update the transformation $T$, aligning the moving image to the fixed image.
• Figure 2: Qualitative comparison of the image registration results. The reference image (first column) is compared with the registered moving image obtained after 2000 iterations (presented in the second and third columns). The registered image obtained using IMPACT is presented in the second column and the one obtained via NCC in the third. Each row presents an example from two distinct patients.
• Figure 3: Qualitative results obtained using VoxelMorph on Task 3 with different loss functions. The first column presents the reference image, followed by results from rigid alignment (Rigid) and the registered CT images estimated using three loss functions: MI, MIND, and the proposed IMPACT loss. Each row illustrates a different example, highlighting the influence of the loss function on registration quality.
• Figure 4: Registration performance measured in terms of TRE. The box plots represent the distribution of errors across varying numbers of iterations, comparing different cost functions: MSE, NMI, NCC, and IMPACT in both Jacobian and Static modes.
• Figure 5: Boxplot of DSC per organ for rigid registration and non-rigid registration using VoxelMorph with different loss functions, including MI, MIND, and IMPACT. The figure also presents DSC values for IMPACT when applied with feature representations from layers 2, 3, and 4 of the decoder (Dec) and encoder (Enc) of the TotalSegmentator model M730, and the second layer of the MedSAM3D network.