SurgPointTransformer: Vertebrae Shape Completion with RGB-D Data

TL;DR

The SurgPointTransformer method enables 3D reconstruction of the entire lumbar spine and surgical guidance without ionizing radiation or invasive imaging and contributes to computer-aided and robot-assisted surgery, advancing the perception and intelligence of these systems.

Abstract

State-of-the-art computer- and robot-assisted surgery systems heavily depend on intraoperative imaging technologies such as CT and fluoroscopy to generate detailed 3D visualization of the patient's anatomy. While imaging techniques are highly accurate, they are based on ionizing radiation and expose patients and clinicians. This study introduces an alternative, radiation-free approach for reconstructing the 3D spine anatomy using RGB-D data. Drawing inspiration from the 3D "mental map" that surgeons form during surgeries, we introduce SurgPointTransformer, a shape completion approach for surgical applications that can accurately reconstruct the unexposed spine regions from sparse observations of the exposed surface. Our method involves two main steps: segmentation and shape completion. The segmentation step includes spinal column localization and segmentation, followed by vertebra-wise segmentation. The segmented vertebra point clouds are then subjected to SurgPointTransformer, which leverages an attention mechanism to learn patterns between visible surface features and the underlying anatomy. For evaluation, we utilize an ex-vivo dataset of nine specimens. Their CT data is used to establish ground truth data that were used to compare to the outputs of our methods. Our method significantly outperforms the state-of-the-art baselines, achieving an average Chamfer Distance of 5.39, an F-Score of 0.85, an Earth Mover's Distance of 0.011, and a Signal-to-Noise Ratio of 22.90 dB. This study demonstrates the potential of our reconstruction method for 3D vertebral shape completion. It enables 3D reconstruction of the entire lumbar spine and surgical guidance without ionizing radiation or invasive imaging. Our work contributes to computer-aided and robot-assisted surgery, advancing the perception and intelligence of these systems.

Figures (5)

• Figure 1: Pipeline of the proposed method for vertebrae shape completion from RGB-D data: $I_{RGB}$ from the RGB-D data is fed into the spinal column detector, which localizes the spine's position with a bounding box $B_{Spine}$ (indicated in fuchsia). $B_{Spine}$ and $I_{RGB}$ are then passed to the spine segmentation model, resulting in a spinal segmentation mask ($M_{Spine}$).This mask then applied to $I_{RGB}$ and $I_{Depth}$ to produce $PCD_{Spine}$. The vertebra-level segmentation module produces color-coded segmentation ($Pred_{Seg}$) for each vertebra level, where red, green, blue, yellow, fuchsia, and black colors correspond to L1, L2, L3, L4, L5, and background, respectively. The segmented vertebra point clouds ($Pred_{Partial}$) are input to our SurgPointTransformer, which reconstructs the complete shape of each vertebra ($Pred_{Complete}$). The completed point clouds are converted into 3D meshes ($Pred_{3D}$).
• Figure 2: Visual Representation of Segmentation and Shape Completion Outputs for L1-L5 Vertebrae from VRCNet and SurgPointTransformer. This figure shows axial, coronal, and sagittal views of shape completion outputs for the L1 (first row) through L5 vertebrae (last row). The outputs from our approach are in fuchsia, and from the state-of-the-art baseline, VRCNet, are in blue. Both are overlaid with the ground truth point cloud in green. The figure also includes evaluation scores for the shape completion results.
• Figure 3: Poisson surface reconstruction kazhdan2006poisson applied on SurgPointTransformer (shown in fuchsia) and VRCNet (shown in blue) outputs overlaid on the ground truth 3D meshes (shown in green) in axial and lateral views.
• Figure 4: Segmentation results for XYZ and XYZRGB inputs are shown with ground truth. Point clouds are overlaid on 3D vertebra meshes, with L1 (red), L2 (green), L3 (blue), L4 (yellow), and L5 (pink) highlighted separately.
• Figure 5: Correlation matrix between variables (specimen and vertebrae level) and evaluation matrices.