Video Dataset for Surgical Phase, Keypoint, and Instrument Recognition in Laparoscopic Surgery (PhaKIR)

TL;DR

The paper introduces the PhaKIR dataset for RAMIS, addressing the need for large, multi-task, temporally coherent datasets collected across multiple centers. It provides eight complete laparoscopic cholecystectomies with unified frame-level annotations for surgical phase recognition, instrument instance segmentation, and instrument keypoint estimation, enabling temporal modeling and cross-task benchmarking. Validation includes a Dice-score of 83.64% for segmentation and a structured 3-tier annotation process, with community-driven evaluation through the MICCAI EndoVis PhaKIR Challenge to ensure robustness. This dataset advances reproducibility and cross-institutional generalization in surgical data science by enabling joint analysis of phase context, instrument pose, and pixel-level localization.

Abstract

Robotic- and computer-assisted minimally invasive surgery (RAMIS) is increasingly relying on computer vision methods for reliable instrument recognition and surgical workflow understanding. Developing such systems often requires large, well-annotated datasets, but existing resources often address isolated tasks, neglect temporal dependencies, or lack multi-center variability. We present the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, comprising eight complete laparoscopic cholecystectomy videos recorded at three medical centers. The dataset provides frame-level annotations for three interconnected tasks: surgical phase recognition (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames). PhaKIR is, to our knowledge, the first multi-institutional dataset to jointly provide phase labels, instrument pose information, and pixel-accurate instrument segmentations, while also enabling the exploitation of temporal context since full surgical procedure sequences are available. It served as the basis for the PhaKIR Challenge as part of the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 to benchmark methods in surgical scene understanding, thereby further validating the dataset's quality and relevance. The dataset is publicly available upon request via the Zenodo platform.

Figures (5)

• Figure 1: Overview of the PhaKIR dataset, illustrating source video data and the annotations for the three tasks: surgical phase recognition, instrument instance segmentation, and instrument keypoint estimation, across three medical centers.
• Figure 2: Visualization of the relative duration of each surgical phase for each video. The phases are arranged in order of their most frequent occurrence across all recorded interventions.
• Figure 3: Instrument types included in the dataset. In case of more than one instrument in an image, the instruments are listed from left to right in the order they appear: Grasper, PE-Forceps (a), Clip-Applicator (b), Scissor (c), Trocar-Tip, Suction-Rod, Palpation-Probe, HF-Coag.-Probe (d), Needle-Probe (e), Argonbeamer (f), Blunt-Grasper-Spec., Bipolar-Clamp (g), Blunt-Grasper, Blunt-Grasper-Curved (h), Blunt-Grasper-Spec., Dissection-Hook, Trocar-Tip (i), Sponge-Clamp (j), Drainage (k), Overholt (l).
• Figure 4: Number of frames in which the respective instrument type occurs.
• Figure 5: Structure of the dataset. The videos available are displayed on the left, the middle column shows the individual elements of each video, and the fine-grained folder structure for the frames and masks is displayed on the right. The elements marked in blue are generated after the frame extraction using the provided script.