Next-generation Surgical Navigation: Marker-less Multi-view 6DoF Pose Estimation of Surgical Instruments

TL;DR

This work investigates marker-less 6DoF pose estimation for surgical instruments using a multi-view RGB-D setup, addressing the gap left by single-view methods. It introduces a comprehensive multi-camera spine surgery dataset with rich annotations and ground-truth, and evaluates single-view and multi-view baselines (ZebraPose, SurfEmb, EpiSurfEmb) across varied camera configurations. The results demonstrate that mm-level accuracy is achievable with as few as two well-placed cameras, and that synthetic data combined with real data yields the best generalization, highlighting practical implications for next-generation surgical navigation. The study also discusses limitations, such as ground-truth bias and the need for in-domain data in challenging lighting conditions, and outlines directions for future improvements in temporal integration and uncertainty estimation to bring marker-less tracking closer to clinical deployment.

Abstract

State-of-the-art research of traditional computer vision is increasingly leveraged in the surgical domain. A particular focus in computer-assisted surgery is to replace marker-based tracking systems for instrument localization with pure image-based 6DoF pose estimation using deep-learning methods. However, state-of-the-art single-view pose estimation methods do not yet meet the accuracy required for surgical navigation. In this context, we investigate the benefits of multi-view setups for highly accurate and occlusion-robust 6DoF pose estimation of surgical instruments and derive recommendations for an ideal camera system that addresses the challenges in the operating room. The contributions of this work are threefold. First, we present a multi-camera capture setup consisting of static and head-mounted cameras, which allows us to study the performance of pose estimation methods under various camera configurations. Second, we publish a multi-view RGB-D video dataset of ex-vivo spine surgeries, captured in a surgical wet lab and a real operating theatre and including rich annotations for surgeon, instrument, and patient anatomy. Third, we evaluate three state-of-the-art single-view and multi-view methods for the task of 6DoF pose estimation of surgical instruments and analyze the influence of camera configurations, training data, and occlusions on the pose accuracy and generalization ability. The best method utilizes five cameras in a multi-view pose optimization and achieves an average position and orientation error of 1.01 mm and 0.89° for a surgical drill as well as 2.79 mm and 3.33° for a screwdriver under optimal conditions. Our results demonstrate that marker-less tracking of surgical instruments is becoming a feasible alternative to existing marker-based systems.

Figures (23)

• Figure 1: Excerpt of our test set in a surgical wet lab (left) and an operating room (right). 6dof pose estimation of surgical instruments is a complex task due to challenging lighting conditions, frequent occlusions, as well as motion blur in ego-centric perspectives. The superimposed outlines indicate the ground truth pose (green) and the pose estimate of our multi-view baseline (red).
• Figure 2: Overview of the camera views in the surgical wet lab setup, with ground truth pose overlays of the drill, anatomy, hand tracking, and eye gaze. The shown cameras are (top-to-bottom, left-to-right) left (L), opposite left (OL), opposite right (OR), right (R), ceiling (C), surgeon (S), and assistant (A).
• Figure 3: Superior and lateral view of an exemplary drill trajectory inside the L4 vertebra.
• Figure 4: Overview of the multi-camera acquisition setup in the surgical wet lab. Multiple static RGB-D cameras are placed around the operating field and on the ceiling. The surgeon and assistant are equipped with hmd. All cameras are calibrated beforehand. To obtain accurate ground truth data, all instruments and hmd are tracked with a marker-based tracking system.
• Figure 5: Both instruments, the hmd, and the anatomy are tracked with a marker-based tracking system. We use hemispherical fiducials with 3mm diameter on instruments and hmd, and spherical fiducials with 12mm diameter for the anatomy.