Markerless Augmented Reality Registration for Surgical Guidance: A Multi-Anatomy Clinical Accuracy Study

TL;DR

The paper tackles fiducial-free augmented reality guidance in surgery by developing a depth-based markerless registration pipeline on the HoloLens 2. It combines depth bias correction, a human-in-the-loop ROI initialization, and a robust coarse-to-fine registration (global TEASER++ followed by ICP) to align preoperative CT skin meshes to intraoperative depth data for multiple anatomies. Clinically, the method achieves a median per-point error around 3–4 mm in live procedures across feet, ear, and lower leg, with high surface coverage and no fiducials, indicating practical viability for moderate-risk tasks. This work advances clinical readiness of markerless AR in surgical navigation and informs design considerations for AR-guided workflows in settings with small or low-curvature targets.

Abstract

Purpose: In this paper, we develop and clinically evaluate a depth-only, markerless augmented reality (AR) registration pipeline on a head-mounted display, and assess accuracy across small or low-curvature anatomies in real-life operative settings. Methods: On HoloLens 2, we align Articulated HAnd Tracking (AHAT) depth to Computed Tomography (CT)-derived skin meshes via (i) depth-bias correction, (ii) brief human-in-the-loop initialization, (iii) global and local registration. We validated the surface-tracing error metric by comparing "skin-to-bone" relative distances to CT ground truth on leg and foot models, using an AR-tracked tool. We then performed seven intraoperative target trials (feet x2, ear x3, leg x2) during the initial stage of fibula free-flap harvest and mandibular reconstruction surgery, and collected 500+ data per trial. Results: Preclinical validation showed tight agreement between AR-traced and CT distances (leg: median |Delta d| 0.78 mm, RMSE 0.97 mm; feet: 0.80 mm, 1.20 mm). Clinically, per-point error had a median of 3.9 mm. Median errors by anatomy were 3.2 mm (feet), 4.3 mm (ear), and 5.3 mm (lower leg), with 5 mm coverage 92-95%, 84-90%, and 72-86%, respectively. Feet vs. lower leg differed significantly (Delta median ~1.1 mm; p < 0.001). Conclusion: A depth-only, markerless AR pipeline on HMDs achieved ~3-4 mm median error across feet, ear, and lower leg in live surgical settings without fiducials, approaching typical clinical error thresholds for moderate-risk tasks. Human-guided initialization plus global-to-local registration enabled accurate alignment on small or low-curvature targets, improving the clinical readiness of markerless AR guidance.

Figures (4)

• Figure 1: Markerless AR registration pipeline. (a) Preoperative CT data is segmented to produce a 3D mesh of the Region of Interest (ROI). Only the skin mesh (feet, lower leg, or ear) is converted to a downsampled point cloud. (b) Intraoperatively, the HoloLens captures a depth image of the exposed anatomy, which is transformed into a point cloud (after correcting sensor bias). The registration module performs a coarse and fine alignment of the preoperative and intraoperative point clouds. (c) We track the handheld tool with fiducials using the HoloLens' depth camera to create a virtual overlay for surface tracing and error measurements, which is evaluated in our experiment.
• Figure 2: Our proposed ROI initialization and cropping process. (a) Mixed reality capture of HoloLens showing both the unregistered target and the translucent reg target when the user first puts on HoloLens. (b) ROI cropping process where the user moves his head to roughly move the translucent red target to the target location. The red target is following head-forward direction constantly. (c) Precise automatic alignment of unregistered target to patient's right foot.
• Figure 3: Relative distance comparison (AR‑traced vs CT). Left: AR‑traced distance mm) from skin to part of internal structures (e.g. fibula/metatarsals). Right: CT ground‑truth distances.
• Figure 4: Distributional characteristics of registration error across anatomies. From left to right: Overlaid histograms, violin plots showing group variance, and coverage curves showing surface coverage.