Extend Your Horizon: A Device-Agnostic Surgical Tool Tracking Framework with Multi-View Optimization for Augmented Reality

TL;DR

This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation and demonstrates improved robustness and enhanced consistency of AR visualization in the presence of occlusions.

Abstract

Surgical navigation provides real-time guidance by estimating the pose of patient anatomy and surgical instruments to visualize relevant intraoperative information. In conventional systems, instruments are typically tracked using fiducial markers and stationary optical tracking systems (OTS). Augmented reality (AR) has further enabled intuitive visualization and motivated tracking using sensors embedded in head-mounted displays (HMDs). However, most existing approaches rely on a clear line of sight, which is difficult to maintain in dynamic operating room environments due to frequent occlusions caused by equipment, surgical tools, and personnel. This work introduces a framework for tracking surgical instruments under occlusion by fusing multiple sensing modalities within a dynamic scene graph representation. The proposed approach integrates tracking systems with different accuracy levels and motion characteristics while estimating tracking reliability in real time. Experimental results demonstrate improved robustness and enhanced consistency of AR visualization in the presence of occlusions.

Figures (6)

• Figure 1: We integrate numerous trackers of diverse types into a standard surgical tracking scenario, ensuring continuity during clinical procedures and preventing track loss. (a) In cases of occlusion, our framework autonomously searches plausible paths within the Dynamic Scene Graph to reestablish tracking continuity. When direct tracking is unattainable, our framework delivers fusion uncertainty estimations, enabling the completion of the AR scene for surgical guidance assistance. (b) The architecture of our framework has a central server that communicates with other sensors through TCP/IP. Each sensor tracks surgical tools independently and updates its tracking results based on the feedback from the central server. AR HMD works as a sensor and a visualization tool to render the tracked tools with their optimized poses.
• Figure 2: (a) A mapping from a mock AR-guided orthopedic surgery to DSG representation. The surgeon with an AR-HMD interacts with a bone phantom positioned on an X-ray table under a C-arm device. Both the phantom and the surgeon's pointer are tagged with retro-reflective markers to enable tracking by OTS and AR-HMD. Within the DSG, passive nodes such as the C-arm, bone model, and pointer are depicted using $n_{pj}, j \in {1,2,3}$ (highlighted in orange), while active nodes like the OTS and AR-HMD are denoted as $n_{ai}, i \in {1,2}$ (in dark blue). The inter-layer edges bridge the $n_{ai}$ and $n_{pj}$, meaning the passive nodes are being tracked by the active nodes. The intra-layer edges connect only $n_{pj}$, representing the relative pose between the passive nodes. (b) demonstrates that when a direct tracking from $n_{a1}$ to $n_{p1}$ is lost, we can still track it indirectly through querying a path within the DSG.
• Figure 3: Example illustrating how our framework compensates for tracking loss. Top: At frame 103, all three objects are directly visible to the HoloLens, and a green indicator is displayed next to each rendered mesh to denote reliable tracking. Bottom: After the pointer is occluded by a hand at frame 115, the green indicator transitions into a yellow uncertainty ellipsoid, which represents the estimated pose uncertainty of the restored object based on the optimized dynamic scene graph. Note that the apparent misalignment between the virtual and physical objects is caused by the mixed reality capture functionality of the HoloLens, and not due to tracking errors.
• Figure 4: (a) Experiment setup for a real-world quantitative experiment. Two operators were wearing HoloLens 2 for tracking and real-time rendering of the tracked markers. An NDI Polaris optical tracker and an Atracsys FusionTrack optical tracker were placed next to the operating table with their view point set to cover the table. The operators used pointers to mimic a drilling procedure on a phantom anatomy. (b) Three types of markers were used in the experiment. Each marker is equipped with four reflective spheres.
• Figure 5: A segment of tracking for the ground truth, the HoloLens tracking, the NDI tracking, and our framework. Top panel shows the tracking status (tracked or lose track), and the bottom panel shows the trajectories decomposed into three axes.