Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction

Qi Wang, Zhijie Wen, Jun Shi, Qian Wang, Dinggang Shen, Shihui Ying

TL;DR

This work tackles the challenge of fast, high-quality cross-modal MRI reconstruction by leveraging fully sampled T1-weighted images to accelerate T2-weighted image acquisitions. It introduces a novel OT-based framework that decomposes cross-modal synthesis into spatial alignment ($OT_{ ext{S}}$) and cross-modal synthesis ($OT_{ ext{M}}$), embedded in an alternating optimization with the reconstruction network. A theoretical bound links reconstruction and synthesis errors, suggesting their mutual reinforcement as iterations proceed, while empirical results on FastMRI and an in-house dataset demonstrate significant improvements at low sampling rates. The approach offers interpretable transport mappings, reduces misalignment-induced degradation, and lowers computational demands relative to competing methods, enhancing practical viability for clinical deployment.

Abstract

Multi-modal magnetic resonance imaging (MRI) plays a crucial role in comprehensive disease diagnosis in clinical medicine. However, acquiring certain modalities, such as T2-weighted images (T2WIs), is time-consuming and prone to be with motion artifacts. It negatively impacts subsequent multi-modal image analysis. To address this issue, we propose an end-to-end deep learning framework that utilizes T1-weighted images (T1WIs) as auxiliary modalities to expedite T2WIs' acquisitions. While image pre-processing is capable of mitigating misalignment, improper parameter selection leads to adverse pre-processing effects, requiring iterative experimentation and adjustment. To overcome this shortage, we employ Optimal Transport (OT) to synthesize T2WIs by aligning T1WIs and performing cross-modal synthesis, effectively mitigating spatial misalignment effects. Furthermore, we adopt an alternating iteration framework between the reconstruction task and the cross-modal synthesis task to optimize the final results. Then, we prove that the reconstructed T2WIs and the synthetic T2WIs become closer on the T2 image manifold with iterations increasing, and further illustrate that the improved reconstruction result enhances the synthesis process, whereas the enhanced synthesis result improves the reconstruction process. Finally, experimental results from FastMRI and internal datasets confirm the effectiveness of our method, demonstrating significant improvements in image reconstruction quality even at low sampling rates.

Spatial and Modal Optimal Transport for Fast Cross-Modal MRI Reconstruction

TL;DR

This work tackles the challenge of fast, high-quality cross-modal MRI reconstruction by leveraging fully sampled T1-weighted images to accelerate T2-weighted image acquisitions. It introduces a novel OT-based framework that decomposes cross-modal synthesis into spatial alignment () and cross-modal synthesis (), embedded in an alternating optimization with the reconstruction network. A theoretical bound links reconstruction and synthesis errors, suggesting their mutual reinforcement as iterations proceed, while empirical results on FastMRI and an in-house dataset demonstrate significant improvements at low sampling rates. The approach offers interpretable transport mappings, reduces misalignment-induced degradation, and lowers computational demands relative to competing methods, enhancing practical viability for clinical deployment.

Abstract

Multi-modal magnetic resonance imaging (MRI) plays a crucial role in comprehensive disease diagnosis in clinical medicine. However, acquiring certain modalities, such as T2-weighted images (T2WIs), is time-consuming and prone to be with motion artifacts. It negatively impacts subsequent multi-modal image analysis. To address this issue, we propose an end-to-end deep learning framework that utilizes T1-weighted images (T1WIs) as auxiliary modalities to expedite T2WIs' acquisitions. While image pre-processing is capable of mitigating misalignment, improper parameter selection leads to adverse pre-processing effects, requiring iterative experimentation and adjustment. To overcome this shortage, we employ Optimal Transport (OT) to synthesize T2WIs by aligning T1WIs and performing cross-modal synthesis, effectively mitigating spatial misalignment effects. Furthermore, we adopt an alternating iteration framework between the reconstruction task and the cross-modal synthesis task to optimize the final results. Then, we prove that the reconstructed T2WIs and the synthetic T2WIs become closer on the T2 image manifold with iterations increasing, and further illustrate that the improved reconstruction result enhances the synthesis process, whereas the enhanced synthesis result improves the reconstruction process. Finally, experimental results from FastMRI and internal datasets confirm the effectiveness of our method, demonstrating significant improvements in image reconstruction quality even at low sampling rates.
Paper Structure (23 sections, 1 theorem, 11 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 23 sections, 1 theorem, 11 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep Learning for Accelerating MRI
  4. Cross-modal Image Synthesis
  5. OT-based Learning Methods
  6. Methods
  7. T1WI-aided Cross-modal Synthesis Process
  8. An OT-view of Spatial Alignment Module
  9. An OT-view of the Cross-modal Synthesis Module
  10. The Complementarity between Two Tasks
  11. The Training Scheme of the Proposed Method
  12. Experiments
  13. Experimental Datasets
  14. Baselines and Implementation Details
  15. Experiments on Two MRI Datasets
  16. ...and 8 more sections

Key Result

Theorem 1

Let $x_{T2}^{R}$, $x_{T2}^{G}$ be the reconstructed T2WI and the synthetic T2WI, and $x_{T2}$ be the full sampled T2WI, then we have the following inequality held: where $\|\cdot\|_{L_1}$ and $\|\cdot\|_{W_1}$ are $L_1$ and $1$-Wasserstein norms, respectively, and the $C>0$ is a constant.

Figures (7)

  • Figure 1: In the zoomed-in views, we observe a slight spatial misalignment between the auxiliary (T1-weighted) image and the target (T2-weighted) image. The paper 8 addresses this issue by using a spatial alignment method to generate the auxiliary alignment. The warped-map illustrates the disparity between the original and warped auxiliary (T1-weighted) images. The color transition from blue to red indicates an increasing difference between the two images.
  • Figure 2: The exhibition of the designed framework. The reconstruction network $R_{\theta}$ uses the under-sampled T2WI $\overline{x}_{T2}$ to obtain the reconstructed T2WI $p(x_{T2}^{R})$ concentrated on $T2$ manifold. The OT process about the spatial alignment module $OT_{\mathcal{S}}$ maps the distribution of the full-sampled T1WI $p(x_{T1})$ to the distribution of the aligned T1WI $p(x_{T1}^A)$ in $T1$ manifold, which is closer to the distribution of the reconstructed T2WI $p(x_{T2}^{R})$. The OT process about the cross-modal synthesis module $OT_{\mathcal{M}}$ transports the distribution of the aligned T1WI $p(x_{T1}^A)$ to the distribution of the synthetic T2WI $p(x_{T2}^G)$ in $T2$ manifold, which is similar to the distribution of the ground truth T2WI $p(x_{T2})$.
  • Figure 3: The provided violin plot visually depicts the results of the Fast dataset obtained through random sampling at a ratio of 6.25% using random masks. The plot showcases three distinct violin-shaped distributions, each representing the outcomes for PSNR, SSIM, and NMSE obtained through different methods.
  • Figure 4: Comparison of the reconstruction results on the FastMRI dataset is shown. The first row shows outcomes from 16× equispaced sampling. The fourth row reveals results from 16× random sampling. Lastly, the seventh row depicts outcomes using 16× radial sampling.
  • Figure 5: Comparison of the reconstruction results on the in-house dataset is shown. The first row shows outcomes from 16× equispaced sampling. The fourth row reveals results from 16× random sampling. Lastly, the seventh row depicts outcomes using 16× radial sampling.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Theorem 1