ViTALS: Vision Transformer for Action Localization in Surgical Nephrectomy

TL;DR

The paper tackles surgical action localization in videos under limited data by introducing the UroSlice nephrectomy dataset and a Vision Transformer–based model, ViTALS. ViTALS combines temporal dilated convolutions with a ViT‑based encoder–decoder architecture and cross-attention, using a ResNet50 spatial feature extractor and a temporal fusion head, trained with a composite loss. On Cholec80 and UroSlice, ViTALS achieves state-of-the-art accuracies of $89.8\%$ and $66.1\%$, respectively, with notable gains on the more challenging UroSlice and robust performance across metrics. Ablation studies confirm the benefit of a ResNet50 FE and multi-stage decoders, underscoring ViTALS’ robustness for small medical datasets and its potential for improved surgical phase recognition and workflow analysis.

Abstract

Surgical action localization is a challenging computer vision problem. While it has promising applications including automated training of surgery procedures, surgical workflow optimization, etc., appropriate model design is pivotal to accomplishing this task. Moreover, the lack of suitable medical datasets adds an additional layer of complexity. To that effect, we introduce a new complex dataset of nephrectomy surgeries called UroSlice. To perform the action localization from these videos, we propose a novel model termed as `ViTALS' (Vision Transformer for Action Localization in Surgical Nephrectomy). Our model incorporates hierarchical dilated temporal convolution layers and inter-layer residual connections to capture the temporal correlations at finer as well as coarser granularities. The proposed approach achieves state-of-the-art performance on Cholec80 and UroSlice datasets (89.8% and 66.1% accuracy, respectively), validating its effectiveness.

Figures (2)

• Figure 1: Overview of our Proposed 'ViTALS' model for surgical phase recognition. The feature extractor model extracts the spatial vector ($e_i$) information from the video sequence ($x_i$). The action segmentation network consists of multi-stage encoder-decoder(s) to perform refined phase prediction. The Encoder module (in red) consisting of a hierarchical network of individual ViT encoder blocks with intermediate residual connections produces an initial prediction. Within each of the encoder blocks, a dilated temporal feed-forward convolution layer is followed by a self-attention layer with residual connections. The Decoder module (in blue) compromises a similar hierarchical network. However, in each decoder block the initial predictions are leveraged for a cross-attention mechanism to refine the granular predictions further.
• Figure 2: Illustrations depicting the qualitative results of surgical phase recognition for (a) Cholec80 and (b) UroSlice datasets are presented. Each line represents the predicted output at different stages of the segmentation network, aligned with the corresponding ground truth labels (GT). Our 'ViTALS' model demonstrates the refinement and produce comparable results to the ground truth predictions for both datasets.