Deep, Deep Learning with BART

TL;DR

To develop a deep‐learning‐based image reconstruction framework for reproducible research in MRI, a network of supervised experiments and real-time measurements were used to demonstrate the power of deep learning in image reconstruction.

Abstract

Purpose: To develop a deep-learning-based image reconstruction framework for reproducible research in MRI. Methods: The BART toolbox offers a rich set of implementations of calibration and reconstruction algorithms for parallel imaging and compressed sensing. In this work, BART was extended by a non-linear operator framework that provides automatic differentiation to allow computation of gradients. Existing MRI-specific operators of BART, such as the non-uniform fast Fourier transform, are directly integrated into this framework and are complemented by common building blocks used in neural networks. To evaluate the use of the framework for advanced deep-learning-based reconstruction, two state-of-the-art unrolled reconstruction networks, namely the Variational Network [1] and MoDL [2], were implemented. Results: State-of-the-art deep image-reconstruction networks can be constructed and trained using BART's gradient based optimization algorithms. The BART implementation achieves a similar performance in terms of training time and reconstruction quality compared to the original implementations based on TensorFlow. Conclusion: By integrating non-linear operators and neural networks into BART, we provide a general framework for deep-learning-based reconstruction in MRI.

Figures (10)

• Figure 1: Integration of deep learning modules into BART. The numerical backend (red) is accessed by md-functions which invoke BART's internal generically-optimized functions or external libraries offering highly optimized code for special functions. Differentiable neural networks are implemented as non-linear operators (blue). The nn-library (green) extends the non-linear operator framework by deep learning specific features. The training algorithms are integrated in BART's iterative framework (violet). Iter6 provides a new interface for batched gradient-based training algorithms.
• Figure 2: Basic concepts of nlops. A) An atomic nlop exemplary with two complex-valued inputs ($\mathbf{x}_1$, $\mathbf{x}_2$) and two outputs ($\mathbf{y}_1=F_1(\mathbf{x}_1,\mathbf{x}_2)$, $\mathbf{y}_2=F_2(\mathbf{x}_1,\mathbf{x}_2)$) consisting of the forward operator$F$ and its derivatives $\mathrm{D}_i{F_o}$ modeled by linops. $F$ and $\mathrm{D}_i{F_o}$ communicate via a shared data structure. B) Chaining of two nlops $F$ and $G$. Since $G$ is applied on the output $F(\mathbf{x})$, its derivative $\mathrm{D}G|_{F(\mathbf{x})}$ is automatically evaluated at $F(\mathbf{x})$. C) The two nlops $F$ and $G$ are combined to form $H$, whose output 1 is linked into input 1 to form $I$, whose inputs 0 and 1 are duplicated to construct $J(\mathbf{x}_1,\mathbf{x}_2)=F(\mathbf{x}_1, G(\mathbf{x}_1, \mathbf{x}_2))$. The derivatives of the final operator are constructed automatically (not shown).
• Figure 3: Schematic description of operators, linops and nlops as data structures in BART. Solid lines mean "points to", dotted lines "points to indirectly" and dashed lines "calls". Colons indicate specific realizations of a data structure, i.e. operator_chain_s is the operator_data_s structure used for chaining operators. Objects required to create the respective structures are marked in red. Other structures and references are created automatically. A) An operator holds a reference to a data structure and a function which is called when the operator is applied. B) A linop holds references to multiple operators such as the forward and adjoint operator which share a common data structure. C) An nlop holds references to the non-linear forward operator and linops modeling the derivatives. The forward operator and linops have access to a shared data structure nlop_data_s. D) The data structure of a chain-operator holds references to the chained operators which are applied sequentially, when the chain-operator is applied.
• Figure 4: Comparison of the TensorFlow and BART implementation of VarNet (A) and MoDL (B). For reference, we also show the results of the adjoint reconstruction $A^H\mathbf{y}$ and an $\ell_1$-Wavelet regularized SENSE reconstruction computed using the BART pics tool. Boxplots are based on PSNR and SSIM of the respective evaluation datasets using the coil sensitivities as foreground mask. This mask explains the discrepancy to the SSIM values given at the reconstructed images.
• Figure 5: Comparison of two example reconstructions with MoDL and VarNet using one set of coil sensitivity maps (usual SENSE) and two sets of coil sensitivity maps (soft-SENSE). The aliased k-space data is simulated by first zero-padding the fully-sampled coil-images and afterwards sub-sampling the k-space by a factor of two before applying the usual sampling pattern (every fourth line and 28 auto calibration lines). The usage of two sets of coil sensitivity maps reduce undersampling artifacts (c.f. arrows) and improves the PSNR and SSIM for VarNet and MoDL.