Unsupervised Representation Learning for 3D MRI Super Resolution with Degradation Adaptation

TL;DR

Experimental results show that the novel unsupervised degradation adaptation network (UDEAN) outperforms state-of-the-art networks with an improvement of up to 0.051/3.52 dB in structural similarity (SSIM)/peak signal-to-noise ratio (PSNR) on two public datasets, thus is a promising solution to the challenges in clinical settings.

Abstract

High-resolution (HR) magnetic resonance imaging is critical in aiding doctors in their diagnoses and image-guided treatments. However, acquiring HR images can be time-consuming and costly. Consequently, deep learning-based super-resolution reconstruction (SRR) has emerged as a promising solution for generating super-resolution (SR) images from low-resolution (LR) images. Unfortunately, training such neural networks requires aligned authentic HR and LR image pairs, which are challenging to obtain due to patient movements during and between image acquisitions. While rigid movements of hard tissues can be corrected with image registration, aligning deformed soft tissues is complex, making it impractical to train neural networks with authentic HR and LR image pairs. Previous studies have focused on SRR using authentic HR images and down-sampled synthetic LR images. However, the difference in degradation representations between synthetic and authentic LR images suppresses the quality of SR images reconstructed from authentic LR images. To address this issue, we propose a novel Unsupervised Degradation Adaptation Network (UDEAN). Our network consists of a degradation learning network and an SRR network. The degradation learning network downsamples the HR images using the degradation representation learned from the misaligned or unpaired LR images. The SRR network then learns the mapping from the down-sampled HR images to the original ones. Experimental results show that our method outperforms state-of-the-art networks and is a promising solution to the challenges in clinical settings.

Figures (8)

• Figure 1: Domain interpretation of differences between supervised and unsupervised SR. A large degradation shift exists between the SR result and desired HR image, which is caused by applying a supervised network, which is pre-trained with synthetic LR images, to authentic LR images with degradation deviating.
• Figure 2: Pipeline of the UDEAN for 3D MRI super-resolution reconstruction. The network is fed with the unpaired or misaligned source group HR image patch $\textbf{Y}_{\textbf{s}}$ and the target group LR patch $\textbf{X}_{\textbf{t}}$ in training. During the inference, only the target group LR image patch $\textbf{X}_{\textbf{t}}$ is fed in the network, and the SR image patch $\textbf{Y}_{\textbf{t}}$ is reconstructed.
• Figure 3: Detailed structures of network components. The encoders are constructed with 6 convolutional layers, each followed by a Leaky ReLU layer. The decoders adopt the TS-RCAN backbone. The VGG network is employed as the discriminator.
• Figure 4: Illustration of the degradation adaptation and feature extraction. Images with similar anatomical structures were selected from different participants in the source group (in the top row) and the target group (in the bottom row). $\textbf{Y}_{\textbf{s}}$ was the HR image from the source group, and $\textbf{Y}_{\textbf{t}}$ was the SR image reconstructed using the LR image ($X_t$) from the target group. $X_{{s}\sim{t}}$ was the LR image degraded from $\textbf{Y}_{\textbf{s}}$ by the degradation learning network, and its image quality was comparable to $X_t$. $f_{s1}$ to $f_{s3}$ and $f_{t1}$ to $f_{t3}$ were examples of the feature maps extracted from $\textbf{Y}_{\textbf{s}}$ and $X_t$, respectively. Highly consistent patterns could be found between the feature maps of the source group and the target group, revealing the effectiveness of the degradation adaptation in the latent feature space.
• Figure 5: Illustration of the degradation adaptation process using t-SNE. (A): The distributions of the feature maps from the source domain ($f_s$) and the target domain ($f_t$) were differentiable before degradation adaptation was performed. (B): After degradation adaptation, the distributions of the feature maps overlapped.