Video-based Surgical Tool-tip and Keypoint Tracking using Multi-frame Context-driven Deep Learning Models

TL;DR

This work tackles automatic tracking of surgical tool keypoints in robotic MIS videos by introducing a multi-frame context-driven segmentation framework that jointly segments keypoint ROIs and localizes their centroids. It combines single-frame and multi-frame segmentation models (SFC/MFCNet) with auxiliary optical-flow and monocular depth maps to exploit temporal and geometric context, achieving state-of-the-art performance on EndoVis'15 ($90\%$ detection, $5.27$ px RMS) and strong results on JIGSAWS ($>91\%$ detection, $<4.2$ px RMS). The method's two-stage design—ROI segmentation followed by blob-centroid localization—yields robust tracking under occlusions and motion blur, with practical inference speeds (≥ $3.4$ FPS on A100). The demonstrated accuracy and efficiency support downstream analyses such as surgical skill assessment and automated safety-zone delineation.

Abstract

Automated tracking of surgical tool keypoints in robotic surgery videos is an essential task for various downstream use cases such as skill assessment, expertise assessment, and the delineation of safety zones. In recent years, the explosion of deep learning for vision applications has led to many works in surgical instrument segmentation, while lesser focus has been on tracking specific tool keypoints, such as tool tips. In this work, we propose a novel, multi-frame context-driven deep learning framework to localize and track tool keypoints in surgical videos. We train and test our models on the annotated frames from the 2015 EndoVis Challenge dataset, resulting in state-of-the-art performance. By leveraging sophisticated deep learning models and multi-frame context, we achieve 90\% keypoint detection accuracy and a localization RMS error of 5.27 pixels. Results on a self-annotated JIGSAWS dataset with more challenging scenarios also show that the proposed multi-frame models can accurately track tool-tip and tool-base keypoints, with ${<}4.2$-pixel RMS error overall. Such a framework paves the way for accurately tracking surgical instrument keypoints, enabling further downstream use cases. Project and dataset webpage: https://tinyurl.com/mfc-tracker

Figures (6)

• Figure 1: Proposed keypoint tracking workflow. We perform tracking by (i) segmenting out keypoint regions (keypoint ROIs) using a deep learning-based segmentation model, and (ii) estimating the centroids of segmented output blobs as tool keypoint locations.
• Figure 2: Multi-frame context (MFC) model design for keypoint ROI segmentation. Predictions of $K$ consecutive frames from a trained single-frame context model are computed. Alongside the segmentation maps, $K{-}1$ optical flow maps and $K$ depth maps are also computed. These maps are passed into MFCNet to aggregate multi-frame context and thus estimate accurate keypoint ROI segmentation predictions.
• Figure 3: (Left) Keypoint annotations provided in du2018articulated. (Right) Self-annotated keypoints on JIGSAWS dataset.
• Figure 4: Comparing model performance on the EndoVis 2015 dataset (with Du et al. 2018 du2018articulated annotations). Showing keypoint localization RMS error (in pixels). Our proposed trackers perform better than du2018articulated, and also better than single-frame context models iglovikov2018ternausnetlong2015fullychen2017rethinkingwang2020deepxie2021segformer. Values for du2018articulated are taken directly from the corresponding paper.
• Figure 5: Keypoint localization results - EndoVis'15 dataset. Keypoint ROI segmentations highlighted with colored mask overlays. Green and white crosses indicate GT and estimated keypoint locations.