Multi-sensor Learning Enables Information Transfer across Different Sensory Data and Augments Multi-modality Imaging

TL;DR

This work addresses the limitation of post hoc fusion in multi-modality imaging by introducing a data-driven multi-sensor learning (MSL) framework that enables inter-modality information transfer between CT and MRI. The core approach combines physics-informed encoders, a cross-domain transformer to extract intra- and inter-modality features, and a conditional decoder with a tunable hybridization parameter $\lambda$ to produce a continuum of MSL images, including CT- and MRI-dominant outputs or mixtures. Key contributions include demonstrating that crossover inter-modality information improves image quality for both CT and MRI, enabling synthesis of a missing modality from sparse data, and enabling locally tunable contrast via spatial maps $\hat{\lambda}$, with robust performance on sparse sampling and domain shifts. The results on brain CT-MRI data show strong cross-domain generalization, potential hardware design implications, and broad applicability to other multi-modality imaging tasks beyond CT-MRI.

Abstract

Multi-modality imaging is widely used in clinical practice and biomedical research to gain a comprehensive understanding of an imaging subject. Currently, multi-modality imaging is accomplished by post hoc fusion of independently reconstructed images under the guidance of mutual information or spatially registered hardware, which limits the accuracy and utility of multi-modality imaging. Here, we investigate a data-driven multi-modality imaging (DMI) strategy for synergetic imaging of CT and MRI. We reveal two distinct types of features in multi-modality imaging, namely intra- and inter-modality features, and present a multi-sensor learning (MSL) framework to utilize the crossover inter-modality features for augmented multi-modality imaging. The MSL imaging approach breaks down the boundaries of traditional imaging modalities and allows for optimal hybridization of CT and MRI, which maximizes the use of sensory data. We showcase the effectiveness of our DMI strategy through synergetic CT-MRI brain imaging. The principle of DMI is quite general and holds enormous potential for various DMI applications across disciplines.

Figures (14)

• Figure 1: Data-driven multi-modality imaging with multi-sensor learning.(A) Conventional approach reconstructs independent images of different modalities and fuses them post hoc without any information transfer across different modalities. (B) The proposed DMI strategy enables synergetic imaging by sharing inter-modality features in a latent representation space. (C) Schematic diagram of our MSL framework, featuring a multi-domain encoder-decoder design and a transformer-based cross-domain interaction module that extracts and transfers crossover inter-modality information between the two modalities. The cross-domain interaction module obtains intra- and inter-modality CT and MRI representations to provide a multi-sensor representation. A sequence of MSL images is generated by transforming the multi-sensor representation with a conditional decoder module.
• Figure 2: Qualitative and quantitative analyses of the MSL images.(A) MSL imaging results of three different samples. Sparse images from conventional reconstruction algorithms, ground truth (GT) images, MSL images, and difference images between the MSL image and GT CT and MRI images are shown. (B) Quantitative evaluation of MSL imaging. The MAE, SSIM, and MI metrics of the MSL image with respect to the GT CT and MRI images at different $\lambda$ are shown in the left and middle panels, respectively. The right panel shows the summation of the paired metrics, which remains almost unchanged as $\lambda$ changes.
• Figure 3: MSL imaging with local contrast tuning.(A) The left panel displays the MSL-CT ($\lambda=0$) and MSL-MRI ($\lambda=1.0$) images for three different samples. The corresponding MSL images generated with six different spatial $\lambda$ distributions are shown in the middle and left panels (where the white and black denote $\lambda$=1 and 0, respectively). (B) The optimized spatial maps of $\lambda$ and the corresponding MSL images for the three samples shown in A are presented.
• Figure 4: MSL imaging results with only MRI sensory data input. Even with only MRI k-space data as input during testing, MSL is able to generate a continuum of images with adjustable hybridization of MRI and CT imaging contents.
• Figure 5: MSL imaging results compared with the Pix2pix image translation method. MSL imaging results compared with the Pix2pix image translation method. The outcomes of MSL from both multi-modality and single-modality inputs and that of Pix2pix are present in different columns.