Rigid Single-Slice-in-Volume registration via rotation-equivariant 2D/3D feature matching

TL;DR

A self-supervised 2D/3D registration approach to match a single 2D slice to the corresponding 3D volume and addresses the dimensionality disparity and establishes correspondences between 2D in-plane and 3D out-of-plane rotation-equivariant features by using group equivariant CNNs.

Abstract

2D to 3D registration is essential in tasks such as diagnosis, surgical navigation, environmental understanding, navigation in robotics, autonomous systems, or augmented reality. In medical imaging, the aim is often to place a 2D image in a 3D volumetric observation to w. Current approaches for rigid single slice in volume registration are limited by requirements such as pose initialization, stacks of adjacent slices, or reliable anatomical landmarks. Here, we propose a self-supervised 2D/3D registration approach to match a single 2D slice to the corresponding 3D volume. The method works in data without anatomical priors such as images of tumors. It addresses the dimensionality disparity and establishes correspondences between 2D in-plane and 3D out-of-plane rotation-equivariant features by using group equivariant CNNs. These rotation-equivariant features are extracted from the 2D query slice and aligned with their 3D counterparts. Results demonstrate the robustness of the proposed slice-in-volume registration on the NSCLC-Radiomics CT and KIRBY21 MRI datasets, attaining an absolute median angle error of less than 2 degrees and a mean-matching feature accuracy of 89% at a tolerance of 3 pixels.

Figures (6)

• Figure 1: SLIV-Reg model: a 2D query slice $\mathbf{Q}$ and a 3D search volume $\mathbf{S}$ are processed to identify candidate points $\mathbf{c}_{Q}$ and $\mathbf{c}_{S}$. At each point in $\mathbf{Q}$ one patch is extracted, and in $\mathbf{S}$ a set of R 2D patches in an equidistante sampling of orientations are extracted with the plane extractor (PE). A pre-trained ReResNet-18 encodes each patch to features $\mathbf{f}_{Q}$ and $\mathbf{f}_{S}$. For each $\mathbf{f}_{Q}$ we find the closest feature vector $\mathbf{f}_{S}$, resulting in a set of candidate matches $\mathbf{M}$. These matches serve as basis for a RANSAC estimate of the 2D slice pose in the 3D volume.
• Figure 2: Data used for method evaluation. First row: tumors in CT volumes of NSCLC patients illustrating the variability of tumor shapes; second row: T1-weighted MRI scans of the brain in the Kirby21 dataset.
• Figure 3: Experiment A: (a) The median absolute angle error for varying directional sampling density of the plane extractor. (b) Qualitative 3D visualization of a registration result, showing the predicted and ground truth slice pose, together with their 2D projections on the right hand side.
• Figure 4: Experiment B: (a) The MMA with pixel thresholds 3/5/10 before (dashed line) and after RANSAC (solid line) for varying directional sampling densities R. (b) 3D visualization of a matching result: $\mathbf{c}_S$ candidate points are shown in blue in the 3D space, $\mathbf{c}_Q$ points are visualized in the yz-plane on the right hand side, and accurate/inaccurate matches are highlighted in green/red (MMA pixel threshold of 3).
• Figure 5: Experiment C: (a) For varying angles, intra-point feature distances are illustrated in orange, while inter-point distances are highlighted in blue, accompanied by standard deviation. (b) Median difference between intra- and inter-point feature distances at given directions, shown for absolute angle differences of 10 to 90 degrees.