PatchMorph: A Stochastic Deep Learning Approach for Unsupervised 3D Brain Image Registration with Small Patches

TL;DR

Experiments on human T1 MRI brain images and marmoset brain images from serial 2-photon tomography affirm PatchMorph's superior performance.

Abstract

We introduce "PatchMorph," an new stochastic deep learning algorithm tailored for unsupervised 3D brain image registration. Unlike other methods, our method uses compact patches of a constant small size to derive solutions that can combine global transformations with local deformations. This approach minimizes the memory footprint of the GPU during training, but also enables us to operate on numerous amounts of randomly overlapping small patches during inference to mitigate image and patch boundary problems. PatchMorph adeptly handles world coordinate transformations between two input images, accommodating variances in attributes such as spacing, array sizes, and orientations. The spatial resolution of patches transitions from coarse to fine, addressing both global and local attributes essential for aligning the images. Each patch offers a unique perspective, together converging towards a comprehensive solution. Experiments on human T1 MRI brain images and marmoset brain images from serial 2-photon tomography affirm PatchMorph's superior performance.

Figures (12)

• Figure 1: This figure displays orthogonal sections from three image stacks $\mathbf{I}_i$, each paired with a world transformation matrix that maps image coordinates to physical world coordinates. PatchMorph utilizes these matrices to accurately sample data from the original arrays. Notably, despite their varied appearances, all arrays in this example depict the same physical image.
• Figure 2: Panel (a): PatchMorph selects patch coordinates $\mathbf{X}_\text{t}$ from the 'working canvas' world coordinate field $\mathbf{X}_\text{ref}$. This canvas is determined by the axis-aligned bounding box enclosing the fixed image $\mathbf{I}_\text{f}$ in physical space. Panel (b): For each iteration at $t>0$, the patch coordinates for the moving image are updated using the resulting displacement field $\mathbf{d}_{\text{out}{(t)}}$ in combination with the affine patch coordinate transformation $\mathbf{T}_{\text{ p}_{(t+1)}}$, setting up the input for the subsequent scale.
• Figure 3: PatchMorph orchestrates a sequence of PatchMorph-blocks from coarse to fine scales. Within each block, it samples small 3D patches from both fixed and moving images, standardizes their spatial resolution, and employs a CNN to estimate and refine their relative displacement.
• Figure 4: This figure displays cross-sections of a 3D image stack, $\mathbf{I}_{\text{f}}$, from which 3D patches $\mathbf{p}_{\text{f}}$ are extracted. Patches are sampled using a coarse-to-fine strategy, ensuring that each finer-resolution patch is contained within its coarser-scale predecessor. The bottom row shows cross sections of the patches. The array size of each patch is consistently maintained across all scales.
• Figure 5: This figure delineates the PatchMorph-Block architecture. It details how the block processes inputs, world coordinates for the fixed image patch $\mathbf{X}_{\text{f}{(t)}}$ and the relative displacement patch $\mathbf{d}_{\text{in}{(t)}}$ from the previous scale ($t>0$), to output the updated relative displacement $\mathbf{d}_{\text{out}{(t)}}$. The block computes world coordinates for the moving patch, extracts and concatenates image patches from the image stacks, and applies a CNN to estimate the local voxel displacement field. The resulting field is then translated back to world coordinates using an affine transformation, combined with the input from the previous iteration to form the final output displacement field $\mathbf{d}_{\text{out}_{(t)}}$.