Surgical Scene Segmentation using a Spike-Driven Video Transformer with Real-Time Potential

TL;DR

<3-5 sentence high-level summary> SpikeSurgSeg introduces a spike-driven video Transformer for surgical scene segmentation, addressing the need for real-time, energy-efficient analysis on non-GPU hardware. The method combines a spike-based encoder (CNN+Spatiotemporal Transformer) with a lightweight segmentation head, trained via surgical-scene masked autoencoding and semantic distillation from SAM2 to overcome data scarcity. It achieves competitive $mIoU$ with state-of-the-art ANN models while delivering substantial reductions in latency and energy consumption, surpassing many foundation-model baselines in time-critical contexts. The work demonstrates the practicality of neuromorphic approaches for deployable intra-operative vision, and outlines future directions for large-scale pretraining and multimodal spike-driven perception.

Abstract

Modern surgical systems increasingly rely on intelligent scene understanding to provide timely situational awareness for enhanced intra-operative safety. Within this pipeline, surgical scene segmentation plays a central role in accurately perceiving operative events. Although recent deep learning models, particularly large-scale foundation models, achieve remarkable segmentation accuracy, their substantial computational demands and power consumption hinder real-time deployment in resource-constrained surgical environments. To address this limitation, we explore the emerging SNN as a promising paradigm for highly efficient surgical intelligence. However, their performance is still constrained by the scarcity of labeled surgical data and the inherently sparse nature of surgical video representations. To this end, we propose \textit{SpikeSurgSeg}, the first spike-driven video Transformer framework tailored for surgical scene segmentation with real-time potential on non-GPU platforms. To address the limited availability of surgical annotations, we introduce a surgical-scene masked autoencoding pretraining strategy for SNNs that enables robust spatiotemporal representation learning via layer-wise tube masking. Building on this pretrained backbone, we further adopt a lightweight spike-driven segmentation head that produces temporally consistent predictions while preserving the low-latency characteristics of SNNs. Extensive experiments on EndoVis18 and our in-house SurgBleed dataset demonstrate that SpikeSurgSeg achieves mIoU comparable to SOTA ANN-based models while reducing inference latency by at least $8\times$. Notably, it delivers over $20\times$ acceleration relative to most foundation-model baselines, underscoring its potential for time-critical surgical scene segmentation.

Figures (9)

• Figure 1: Motivation of Our Work. Intelligent surgical systems should achieve both high accuracy and low latency to be deployable in the operating room. Although ANN-based models, especially foundation models, deliver strong performance for surgical scene segmentation, they rely on power-hungry GPUs to maintain low latency, which is impractical under the physical and safety constraints of surgical environments. In contrast, emerging SNN-based models offer inherent efficiency but often suffer from performance degradation due to their sparse spike representations. To overcome this limitation, we propose a surgical-scene masked pretraining strategy that enhances surgical scene segmentation performance of SNNs while retaining their low-latency advantage.
• Figure 2: SpikeSurgSeg Pipeline. 1) Stage 1: Spike-driven video encoder is pretrained on surgical videos using a masked autoencoding strategy, where a layer-wise tube masking is applied to the input video clip. The masked patches are then reconstructed by aggregating information from unmasked regions. 2) Stage 2: Spike-driven memory readout and feature pyramid network are integrated with the spike video encoder, and then the entire SNN-based model is finetuned for the downstream surgical scene segmentation task. The bottom block shows the detailed architecture of the encoder, which employs two spike-driven CNN blocks and two spike-driven spatiotemporal Transformers featuring linear space–time computational complexity.
• Figure 3: Layer-wise Tube Masking. The tube-shaped mask is applied at each encoder layer to prevent any spike generation within the masked areas, ensuring that no spatial or temporal information leaks into the reconstruction process.
• Figure 4: Qualitative Results on EndoVis18 Dataset. Our SNN-based model maintains high efficiency while reducing false positives and inference time, benefiting from the inherent sparsity of spike-driven computation.
• Figure 5: Qualitative Results on SurgBleed Dataset. Compared with the task-specific MATIS-Frame method and the prompt-based SAM2-Finetune method, our SNN-based model achieves comparable or even superior bleeding segmentation performance, while maintaining substantially lower energy consumption and inference latency.