Recognize Any Surgical Object: Unleashing the Power of Weakly-Supervised Data

TL;DR

The paper presents RASO, a foundation model for recognizing any surgical object in images and videos under open-set conditions. It introduces a weakly supervised data generation pipeline and a temporal-attention fusion mechanism built on RAM to scale recognition without exhaustive annotations. RASO achieves state-of-the-art zero-shot and supervised performance on multiple surgical benchmarks and demonstrates efficient training and inference on 8 NVIDIA A6000 GPUs. This work enables broader surgical scene understanding and lays groundwork for integrated recognition-segmentation systems in computer-assisted interventions.

Abstract

We present RASO, a foundation model designed to Recognize Any Surgical Object, offering robust open-set recognition capabilities across a broad range of surgical procedures and object classes, in both surgical images and videos. RASO leverages a novel weakly-supervised learning framework that generates tag-image-text pairs automatically from large-scale unannotated surgical lecture videos, significantly reducing the need for manual annotations. Our scalable data generation pipeline gathers 2,200 surgical procedures and produces 3.6 million tag annotations across 2,066 unique surgical tags. Our experiments show that RASO achieves improvements of 2.9 mAP, 4.5 mAP, 10.6 mAP, and 7.2 mAP on four standard surgical benchmarks, respectively, in zero-shot settings, and surpasses state-of-the-art models in supervised surgical action recognition tasks. Code, model, and demo are available at https://ntlm1686.github.io/raso.

Figures (4)

• Figure 1: Comparison of surgical object recognition performance across different models and clinician annotations. Green for correct tags. Red for wrong tags. Yellow for tags are not accurate but somewhat related. Grey for tags are missing. RASO demonstrates high accuracy in identifying relevant surgical instruments and anatomical structures, aligning closely with clinician annotations.
• Figure 2: RASO Architecture. RASO includes (1) an image encoder to extract visual features from images and video frames; (2) a temporal-attention fusion layer to handle video inputs by capturing temporal dependencies across frames; (3) a tag decoder to predict surgical labels by combining visual and tag embeddings; and (4) a text decoder to align the visual content with transcriptions.
• Figure 3: The weakly supervised pretraining data pipeline (top-left). Surgical lecture videos, including both visual frames and voiceover transcripts, are processed through named entity recognition, semantic parsing, and manual filtering to extract relevant surgical tags. These tags encompass anatomical structures, instruments, and procedural information. The bottom figure gives an overview of the tag frequency in our pretraining dataset using a word cloud.
• Figure 4: Comparison on Video Inference.