A Plug-and-Play Method for Guided Multi-contrast MRI Reconstruction based on Content/Style Modeling

TL;DR

A modular two-stage approach that does not require any k-space training data, relying solely on image-domain datasets, and provides an explanatory framework for the multi-contrast problem based on the shared and non-shared generative factors underlying two given contrasts.

Abstract

Since multiple MRI contrasts of the same anatomy contain redundant information, one contrast can guide the reconstruction of an undersampled subsequent contrast. To this end, several end-to-end learning-based guided reconstruction methods have been proposed. However, a key challenge is the requirement of large paired training datasets comprising raw data and aligned reference images. We propose a modular two-stage approach that does not require any k-space training data, relying solely on image-domain datasets, a large part of which can be unpaired. Additionally, our approach provides an explanatory framework for the multi-contrast problem based on the shared and non-shared generative factors underlying two given contrasts. A content/style model of two-contrast image data is learned from a largely unpaired image-domain dataset and is subsequently applied as a plug-and-play operator in iterative reconstruction. The disentanglement of content and style allows explicit representation of contrast-independent and contrast-specific factors. Consequently, incorporating prior information into the reconstruction reduces to a simple replacement of the aliased content of the reconstruction iterate with high-quality content derived from the reference scan. Combining this component with a data consistency step and introducing a general corrective process for the content yields an iterative scheme. We name this novel approach PnP-CoSMo. Various aspects like interpretability and convergence are explored via simulations. Furthermore, its practicality is demonstrated on the public NYU fastMRI DICOM dataset, showing improved generalizability compared to end-to-end methods, and on two in-house multi-coil raw datasets, offering up to 32.6\% more acceleration over learning-based non-guided reconstruction for a given SSIM.

Figures (14)

• Figure 1: Our two-stage approach to guided reconstruction. (a) The first stage learns a content/style model of two-contrast MR image data offline. No k-space data is required in this stage. We assume that the two image domains $\mathcal{X}_1$ and $\mathcal{X}_2$ can be decomposed into a shared content domain $\mathcal{C}$ and separate style domains $\mathcal{S}_1$ and $\mathcal{S}_2$. This model is learned in two stages -- an unpaired pre-training stage and a paired fine-tuning stage, both requiring only image data. (b) The reconstruction stage applies the content/style model as a content consistency operator (bottom) within an ISTA-based iterative scheme. Given an aligned reference image, guidance is introduced into the reconstruction by simply replacing its aliased content with content derived from the reference. The "refine" block denotes a content refinement update, which iteratively corrects for inconsistencies between the reference content and the measured k-space data, improving the effectiveness of the content consistency operator. DC denotes data consistency. For comparison, two other reconstruction priors are shown, namely wavelet-domain soft-thresholding (top) and CNN-based denoising (middle) used in CS and PnP-CNN algorithms, respectively.
• Figure 2: Our proposed paired fine-tuning (PFT) stage for improving the alignment of the content representations of the two domains given a small dataset of paired images.
• Figure 3: Demonstration of the content consistency operator $g_M(\cdot;c)$ based on a content/style model $M$. The image pair shown here are NYU fastMRI brain DICOMs, practical details regarding which are provided in Section \ref{['expt_design']}. (a) A T1W/T2W image pair and the corresponding content maps showing that the two contents generally agree while there is also some discrepancy between them. (b) Synthetic T2W images generated from 16 style codes grid-sampled from $\mathcal{S}_2$, showing that style encodes mostly the image contrast. (c) Ground truth T2W $x_2^*$ is corrupted to $x_2^\text{us}$ using $R$=4 undersampling with three different center fractions and the operator $g_M(\cdot;c)$ is applied to each case, showing that sampling more low-frequency lines leads to more accurate style estimate $\hat{s}_2$ and hence a better content-consistent image $x_2^\mathrm{cc}$. Contours in $\mathcal{S}_2$ indicate the level set of MAE of the synthetic images corresponding to styles from those regions. The red circle indicates a style estimation NMSE of 0.1.
• Figure 4: Convergence analysis on synthetic BrainWeb data. (a) Convergence curves for PnP-CoSMo and its two variants at $R$=2. (b) Evolution of the reconstruction shown with the corresponding error maps. In PnP-CoSMo with CR disabled and its upperbound version, i.e. both cases where the content stays constant, the style converges in a single iteration and so does the reconstruction. However, there is a notable discrepancy in the reference content which is reflected in the reconstruction error when CR is disabled. Enabling CR leads to an iterative correction of the content discrepancy, leading in turn to the reconstruction approaching the upperbound quality.
• Figure 5: Lesion analysis on synthetic BrainWeb data. Given a contrast-specific structure in the T2W data such as the simulated lesion here which is absent in the reference image, our CR module contributes substantially in resolving it.