Pixel-Wise Recognition for Holistic Surgical Scene Understanding

TL;DR

GraSP introduces a holistic, multi-granular surgical scene understanding benchmark for robot-assisted prostatectomies and pairs it with TAPIS, a fully transformer-based architecture that jointly handles instrument segmentation, phase/step recognition, and atomic action detection. By leveraging a global video feature extractor (MViT) and region proposals from instrument segmentation (Mask2Former) with a cross-attention region head, TAPIS achieves state-of-the-art results across GraSP and generalizes to public benchmarks like EndoVis 2018 and MISAW. The dataset extends prior PSI-AVA with pixel-level instrument segmentation and tenacious expert-validated annotations, totaling over 32 hours of in vivo data and a MIT license release. Collectively, the work advances holistic surgical scene understanding, enabling more robust context-aware robotic assistance, training, and education in Endoscopic Vision.

Abstract

This paper presents the Holistic and Multi-Granular Surgical Scene Understanding of Prostatectomies (GraSP) dataset, a curated benchmark that models surgical scene understanding as a hierarchy of complementary tasks with varying levels of granularity. Our approach enables a multi-level comprehension of surgical activities, encompassing long-term tasks such as surgical phases and steps recognition and short-term tasks including surgical instrument segmentation and atomic visual actions detection. To exploit our proposed benchmark, we introduce the Transformers for Actions, Phases, Steps, and Instrument Segmentation (TAPIS) model, a general architecture that combines a global video feature extractor with localized region proposals from an instrument segmentation model to tackle the multi-granularity of our benchmark. Through extensive experimentation, we demonstrate the impact of including segmentation annotations in short-term recognition tasks, highlight the varying granularity requirements of each task, and establish TAPIS's superiority over previously proposed baselines and conventional CNN-based models. Additionally, we validate the robustness of our method across multiple public benchmarks, confirming the reliability and applicability of our dataset. This work represents a significant step forward in Endoscopic Vision, offering a novel and comprehensive framework for future research towards a holistic understanding of surgical procedures.

Figures (33)

• Figure 1: The GraSP Dataset formulates a holistic understanding of robot-assisted Radical Prostatectomy videos by studying four hierarchical tasks annotated in their highest granularity. These tasks include two long-term tasks, recognition of surgical phases and steps, and two short-term tasks, surgical instrument segmentation and atomic action detection. Figure best viewed in color.
• Figure 2: Distribution of the percentage of frames per surgical phase category on each data split. We count the number of frames sampled at 1fps. We organize the graphic in descending order and present correspondence between phase IDs and phase labels in Table \ref{['tab:categories']}. The figure is best viewed in color.
• Figure 3: Distribution of the percentage of frames per surgical step category on each data split. We count the number of frames sampled at 1fps. We organize the graphic in descending order and present correspondence between step IDs and step labels in Table \ref{['tab:categories']}. The figure is best viewed in color.
• Figure 4: Boxplot distribution of the duration of each phase category. Each boxplot presents the distribution in seconds of the period of all the present temporal windows corresponding to each phase category in our dataset. We clipped this figure to a maximum of 800 seconds for better visualization; the entire figure is presented in the Supplementary Material. The figure is best viewed in color.
• Figure 5: Boxplot distribution of the duration of each step category. Each boxplot presents the distribution in seconds of the period of all the present temporal windows corresponding to each step category in our dataset. We clipped this figure to a maximum of 800 seconds for better visualization; the entire figure is presented in the Supplementary Material. The figure is best viewed in color.