Semi-Automatic Infrared Calibration for Augmented Reality Systems in Surgery

TL;DR

This work addresses the calibration gap between AR headsets and surgical navigation systems by introducing an IR marker-array–based, fast, user-agnostic registration between a HoloLens 2 and a CAOS optical tracker. It leverages onboard AB/ToF sensing to detect IR markers, reconstruct 3D marker positions, and compute a rigid transformation $T^H_C = T^H_P \cdot (T^C_P)^{-1}$ to align holographic content with the robotic tracking frame. Quantitative tests show mean translation errors around $2.0$ mm and rotation errors around $1.1$–$1.54^{\circ}$ for both relative-tracking and AR-guided tasks, indicating feasibility of AR-assisted CAOS workflows with room for improvement toward clinical thresholds. The approach offers a plug-and-play calibration path that minimizes workflow disruption and avoids introducing new markers or hardware, supporting future mixed-reality surgical workflows.

Abstract

Augmented reality (AR) has the potential to improve the immersion and efficiency of computer-assisted orthopaedic surgery (CAOS) by allowing surgeons to maintain focus on the operating site rather than external displays in the operating theatre. Successful deployment of AR to CAOS requires a calibration that can accurately calculate the spatial relationship between real and holographic objects. Several studies attempt this calibration through manual alignment or with additional fiducial markers in the surgical scene. We propose a calibration system that offers a direct method for the calibration of AR head-mounted displays (HMDs) with CAOS systems, by using infrared-reflective marker-arrays widely used in CAOS. In our fast, user-agnostic setup, a HoloLens 2 detected the pose of marker arrays using infrared response and time-of-flight depth obtained through sensors onboard the HMD. Registration with a commercially available CAOS system was achieved when an IR marker-array was visible to both devices. Study tests found relative-tracking mean errors of 2.03 mm and 1.12° when calculating the relative pose between two static marker-arrays at short ranges. When using the calibration result to provide in-situ holographic guidance for a simulated wire-insertion task, a pre-clinical test reported mean errors of 2.07 mm and 1.54° when compared to a pre-planned trajectory.

Figures (7)

• Figure 1: Comparison of workflows designed for computer and robot-assisted patellofemoral arthroplasty (PFA). Left: screenshots of the standard, monitor-based workflow run by the NAVIO® robot -- a commercial CAOS system. Right: Screenshots of our novel AR-centric workflow which reproduces the NAVIO® robot's entire surgical workflow for PFA in a 3D AR environment through an HMD Iqbal2022AugmentedSurgery -- an application of the AR-registration method described in this paper.
• Figure 2: Pipeline for image processing of sensor images for tool detection and tool pose estimation. Left: Scene prior to tool detection. (1) Input active-brightness (AB) image. (2) Input depth image. (3) Processed AB image following thresholding & blob detection. (4) Reconstructed point cloud from depth image in Figure \ref{['fig:img_proc']}.2, points coloured by IR response. (5) Outlier detection of blobs after unmapping pixel-coordinates to 3D-coordinates and comparing with known tool geometries. (6) Registration process to calculate pose of tool with respect to virtual world frame. Right: Scene after tool detection and AR augmentation.
• Figure 3: Various transformation matrices generated when using system: tool pose in fixed virtual world frame, $T^H_P$, tool pose with respect to surgical robot's optical tracker, $T^C_P$, and calibration matrix which maps between optical tracker and virtual coordinates, $T^H_C$. $T^P_Q$, matrix describing relative pose between two IR marker-arrays.
• Figure 4: Flowchart illustrating how the system can track tools equipped with IR-reflective markers.
• Figure 5: Diagram of system-flow between a HoloLens 2 headset and a custom build of the NAVIO® robot's software