A Multi-View Pipeline and Benchmark Dataset for 3D Hand Pose Estimation in Surgery

TL;DR

The paper tackles robust 3D hand pose estimation in challenging surgical environments by introducing a training-free multi-view pipeline that composes reliable detection, full-body pose estimation, and high-resolution 2D hand refinement, followed by constrained 3D optimization. It contributes a novel multi-view surgical dataset with over 68k frames and 3k annotated 2D hand poses triangulated to 3D ground truth, plus extensive experiments and ablations comparing against strong baselines. The approach achieves substantial accuracy gains (notably 2D MPJPE reduced by ~31% and 3D MPJPE by ~76% versus baselines) and analyzes the impact of loss terms like temporal smoothness and shape constraints. This work establishes a solid training-free baseline for surgical hand pose estimation and provides a valuable annotated resource to accelerate future research in surgical computer vision, with clear paths toward real-time, learning-based extensions.

Abstract

Purpose: Accurate 3D hand pose estimation supports surgical applications such as skill assessment, robot-assisted interventions, and geometry-aware workflow analysis. However, surgical environments pose severe challenges, including intense and localized lighting, frequent occlusions by instruments or staff, and uniform hand appearance due to gloves, combined with a scarcity of annotated datasets for reliable model training. Method: We propose a robust multi-view pipeline for 3D hand pose estimation in surgical contexts that requires no domain-specific fine-tuning and relies solely on off-the-shelf pretrained models. The pipeline integrates reliable person detection, whole-body pose estimation, and state-of-the-art 2D hand keypoint prediction on tracked hand crops, followed by a constrained 3D optimization. In addition, we introduce a novel surgical benchmark dataset comprising over 68,000 frames and 3,000 manually annotated 2D hand poses with triangulated 3D ground truth, recorded in a replica operating room under varying levels of scene complexity. Results: Quantitative experiments demonstrate that our method consistently outperforms baselines, achieving a 31% reduction in 2D mean joint error and a 76% reduction in 3D mean per-joint position error. Conclusion: Our work establishes a strong baseline for 3D hand pose estimation in surgery, providing both a training-free pipeline and a comprehensive annotated dataset to facilitate future research in surgical computer vision.

Figures (7)

• Figure 1: Example showing the challenges of hand localization in surgical scenes caused by lighting conditions and the similar appearance of gloves and gowns. Left: RTMPose jiang2023rtmposerealtimemultipersonpose fails to accurately localize the hands. Middle: our 2D predictions. Right: SMPL-X SMPL-X:2019 3D reconstruction using our 2D predictions from the same viewpoints.
• Figure 2: Overview of our 3D hand pose estimation framework. Given multi-view videos from the OR, we first detect and track persons, then predict full body and refined hand poses. Finally, we optimize a sequence of 3D hand poses consistent with anatomical and temporal constraints.
• Figure 3: Qualitative comparison of 2D hand pose predictions. RTM, DW, and RTM $\rightarrow$ HM baselines are shown alongside our method, which achieves more complete and stable hand poses.
• Figure 4: Qualitative results of the full pipeline showing 3D poses reprojected into the original video streams and a 3D SMPL-X SMPL-X:2019 render (right). First row: SPP. Second and third row: SPI.
• Figure 5: Inter-annotator variability across 2D and 3D annotations. (a) 2D pixel-space variability aggregated across all eight cameras. (b) Mean 3D pairwise distances between all annotator pairs, computed per joint and averaged across all sequences and frames. (c) Illustration of the 21-joint hand model used for labeling and distance computation.