Seamless Augmented Reality Integration in Arthroscopy: A Pipeline for Articular Reconstruction and Guidance

TL;DR

A robust pipeline that incorporates simultaneous localization and mapping, depth estimation, and 3D Gaussian splatting is presented to realistically reconstruct intra‐articular structures solely based on monocular arthroscope video to enhance intraoperative awareness and facilitate surgical precision in arthroscopy.

Abstract

Arthroscopy is a minimally invasive surgical procedure used to diagnose and treat joint problems. The clinical workflow of arthroscopy typically involves inserting an arthroscope into the joint through a small incision, during which surgeons navigate and operate largely by relying on their visual assessment through the arthroscope. However, the arthroscope's restricted field of view and lack of depth perception pose challenges in navigating complex articular structures and achieving surgical precision during procedures. Aiming at enhancing intraoperative awareness, we present a robust pipeline that incorporates simultaneous localization and mapping, depth estimation, and 3D Gaussian splatting to realistically reconstruct intra-articular structures solely based on monocular arthroscope video. Extending 3D reconstruction to Augmented Reality (AR) applications, our solution offers AR assistance for articular notch measurement and annotation anchoring in a human-in-the-loop manner. Compared to traditional Structure-from-Motion and Neural Radiance Field-based methods, our pipeline achieves dense 3D reconstruction and competitive rendering fidelity with explicit 3D representation in 7 minutes on average. When evaluated on four phantom datasets, our method achieves RMSE = 2.21mm reconstruction error, PSNR = 32.86 and SSIM = 0.89 on average. Because our pipeline enables AR reconstruction and guidance directly from monocular arthroscopy without any additional data and/or hardware, our solution may hold the potential for enhancing intraoperative awareness and facilitating surgical precision in arthroscopy. Our AR measurement tool achieves accuracy within 1.59 +/- 1.81mm and the AR annotation tool achieves a mIoU of 0.721.

Figures (5)

• Figure 1: Illustration of the proposed pipeline for reconstruction and AR guidance in Arthroscopy. (a) We reconstruct sparse 3D priors with OneSLAM on an arthroscope footage. For each keyframe, the depth estimation model generates pseudo depth map, the relative scales are recovered for each frame for consistency, normals are generated from a depth-to-normal translator. (b) The sparse point cloud is used to initialize the location of 3D Gaussians, and RGBD images along with normals are taken as supervision for 3D GS model training. (c) In our AR application, we have designed an annotation tool to highlight anatomical structures and a measurement tool to conveniently estimate surface distances.
• Figure 2: The physical setup for phantom data collection. Fiducial markers are rigidly attached to the arthroscope and arthroscopy phantom. The 6 DoF poses are tracked by an optical tracker during inspection. The CT scan is taken after data collection to gain a ground truth model.
• Figure 3: Evaluation results for AR Measurement accuracy. (a) Distribution alignment between the measured distance on our reconstruction (orange), and the ground truth distance (blue) for the evaluation trials. (b) Comparison of the distribution shapes reflecting measurement difference between our method (blue) and Dust3R (orange). (c) Comparison of the key statistical indicators.
• Figure 4: Evaluation results for AR annotation accuracy. (a) Qualitative results for AR annotation accuracy. The annotation masks are highlighted and superimposed on the arthroscopic scene with the corresponding IoU score on top of each method. Our pipeline is comparable with the SOTA segmentation method on initial frames and outperforms it for consecutive frames. (b) The histogram of IoU score for AR annotation and Cutie segmentation. The results of Cutie (orange) are polarized, showing good segmentation on some frames but losing tracking on others. In contrast, our method (blue) achieves relatively steady and consistent annotation.
• Figure 5: Visualization of rendering quality comparison.