Structurally Consistent MRI Colorization using Cross-modal Fusion Learning

TL;DR

The paper tackles the problem of colorizing MRI scans by transferring Cryosection color distributions while preserving the grayscale MRI structure, even with only partial registration between modalities. It introduces a cross-modality, semi-supervised framework with a dual-decoder colorization generator, complemented by compression-activation skip connections and a multiscale module to handle varying resolutions. The training objective combines cyclic adversarial losses, multi-scale SSIM-based structural fidelity losses, and segmentation-guided constraints via a fixed Cryosection segmentation model, enabling organ-aware color and texture transfer. The approach yields state-of-the-art quantitative metrics and convincing qualitative results, demonstrating improved structural integrity and Cryosection-like colorization with practical implications for medical visualization and interpretability.

Abstract

Medical image colorization can greatly enhance the interpretability of the underlying imaging modality and provide insights into human anatomy. The objective of medical image colorization is to transfer a diverse spectrum of colors distributed across human anatomy from Cryosection data to source MRI data while retaining the structures of the MRI. To achieve this, we propose a novel architecture for structurally consistent color transfer to the source MRI data. Our architecture fuses segmentation semantics of Cryosection images for stable contextual colorization of various organs in MRI images. For colorization, we neither require precise registration between MRI and Cryosection images, nor segmentation of MRI images. Additionally, our architecture incorporates a feature compression-and-activation mechanism to capture organ-level global information and suppress noise, enabling the distinction of organ-specific data in MRI scans for more accurate and realistic organ-specific colorization. Our experiments demonstrate that our architecture surpasses the existing methods and yields better quantitative and qualitative results.

Figures (6)

• Figure 1: Architecture of our cycle consistent MRI colorization training model. The primary generator module $\mathcal{G}_{m\rightarrow \hat{c}}$ produces colorized MRI and an auxiliary output of pseudo Cryosection image which is used to build organ level context in the network. We use a pre-trained segmentation network $\mathcal{S}_{c\rightarrow \hat{s}}$. Various losses are indicated with light dashed lines.
• Figure 2: MRI colorization results on three scales of resolution.
• Figure 3: Quantitative comparison of MRI colorization with different methods.
• Figure 4: Zoom-in comparison with competing methods: ColorFormer ji2022colorformer and APS 9761546 produce appealing results, however a closer examination reveals structural dissimilarities with the input MRI and inconsistent colors compared to the ground truth Cryosection.
• Figure 5: Qualitative comparison of the effectiveness of various components of our architecture.