Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Phase Grounding to Intelligent Surgical Narratives

Ethan Peterson, Huixin Zhan

TL;DR

This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.

Abstract

Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.

From Phase Grounding to Intelligent Surgical Narratives

TL;DR

This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.

Abstract

Video surgery timelines are an important part of tool-assisted surgeries, as they allow surgeons to quickly focus on key parts of the procedure. Current methods involve the surgeon filling out a post-operation (OP) report, which is often vague, or manually annotating the surgical videos, which is highly time-consuming. Our proposed method sits between these two extremes: we aim to automatically create a surgical timeline and narrative directly from the surgical video. To achieve this, we employ a CLIP-based multi-modal framework that aligns surgical video frames with textual gesture descriptions. Specifically, we use the CLIP visual encoder to extract representations from surgical video frames and the text encoder to embed the corresponding gesture sentences into a shared embedding space. We then fine-tune the model to improve the alignment between video gestures and textual tokens. Once trained, the model predicts gestures and phases for video frames, enabling the construction of a structured surgical timeline. This approach leverages pretrained multi-modal representations to bridge visual gestures and textual narratives, reducing the need for manual video review and annotation by surgeons.
Paper Structure (15 sections, 5 figures, 6 tables)

This paper contains 15 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Baseline Model
  5. Datasets
  6. Loss Function
  7. Fine-Tuning the CLIP Model on Gestures
  8. Further Fine-Tuning the CLIP Model on Surgical Phases
  9. Experimental Results
  10. Results of Training the Linear Probe
  11. Results of Fine-Tuning the CLIP Model on Gestures
  12. Results of Further Fine-Tuning the CLIP Model on Surgical Phases
  13. Results of Further Fine-Tuning the Base Model
  14. Limitations and Future Work
  15. Conclusion

Figures (5)

  • Figure 1: An Example Frame of the JIGSAWS Dataset gao2014jigsaws (On the Right), and the Cholec80 Dataset twinanda2016endonetdeeparchitecturerecognition (On the Left)
  • Figure 2: CLIP Framework Aligning Video Frames and Text in a Shared Embedding Space, Adapted from radford2021learningtransferablevisualmodels
  • Figure 3: Comparison of Gesture Accuracy Between Base Model and Model Fine-Tuned on JIGSAWS
  • Figure 4: Confusion Matrix with the Colors Normalized
  • Figure 5: Model Prediction vs Ground Truth