Temporally Guided Articulated Hand Pose Tracking in Surgical Videos

TL;DR

Articulated hand pose tracking in surgical videos is challenging due to occlusion and rapid motion. The authors introduce CondPose, a temporally guided pose estimator that conditions current hand predictions on prior frame heatmaps via an attention-based fusion mechanism, and they release Surgical Hands, the first multi-instance hand pose tracking dataset in surgery. Across surgical data and PoseTrack18, CondPose achieves higher mAP for detection and higher MOTA for tracking, with ablations confirming the usefulness of temporal priors for localization. The work provides a practical route to more accurate hand representations in surgery, with potential downstream benefits for skill assessment and action understanding.

Abstract

Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks. In this work, we propose a novel hand pose estimation model, CondPose, which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions. We collect Surgical Hands, the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands, we show our method outperforms the state-of-the-art approach using mean Average Precision (mAP), to measure pose estimation accuracy, and Multiple Object Tracking Accuracy (MOTA), to assess pose tracking performance. In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks.

Figures (6)

• Figure 1: On the left, a method only performing frame-wise independent predictions may miss out on properly localizing joints, while on the right, temporally passing past predictions from previous frames improves the network's localization.
• Figure 2: The baseline generates a heatmap, $\hat{\mathcal{H}'_t}$, for each detection using a pose estimation network. In our model, we provide additional information by incorporating a heatmap prior from $t - \delta$. Concatenating the image features at $t$ with $\hat{\mathcal{H}}_{t-\delta}$, we pass this through our attention mechanism to produce a weighted heatmap prior, $\hat{\mathcal{H}}'_{t-\delta}$. Both $\hat{\mathcal{H}'_t}$ and $\hat{\mathcal{H}}'_{t-\delta}$ are concatenated and passed through the fusing module, using context from both heatmaps to produce the final articulated hand pose. (The initial and final heatmaps represent real outputs from the network, while the heatmap prior (during training) shows ground truth at $t - \delta$)
• Figure 3: We show samples from our annotations. Each hand is labeled with a bounding box, handedness, tracking id, and visibility of joints.
• Figure 4: We show a qualitative comparison between the baseline model and our method. We note a higher recall and consistency between frames, as shown for the hand to the left. Even when the pinky finger is not visible, the past predictions reinforces those joint locations.
• Figure 5: We show qualitative samples of frames from the best performing (top row) and lower performing (bottom row) videos. (Best viewed in color).