Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

Hugo Georgenthum, Cristian Cosentino, Fabrizio Marozzo, Pietro Liò

TL;DR

This work tackles automatic surgical video documentation by presenting a cohesive multimodal pipeline that fuses visual transformer-based feature extraction, frame- and clip-level captioning, and LLM-driven report synthesis. The method decomposes the task into object detection, caption generation, and narrative assembly, leveraging ViT, ViViT, DistilBERT, T5, and GPT-4 to produce structured surgical reports from laparoscopic videos. Through evaluation on the CholecT50 dataset, the approach achieves strong tool detection with calibrated probabilities, robust frame-captioning under noisy inputs, and coherent clip-level descriptions that feed into a final report—demonstrating the feasibility of end-to-end, explainable surgical documentation. The results underscore the practical impact of integrated multimodal reasoning for clinical documentation, training, and retrospective analysis, while outlining avenues for broader procedure coverage, longer temporal contexts, and multi-modal enhancements.

Abstract

The automatic summarization of surgical videos is essential for enhancing procedural documentation, supporting surgical training, and facilitating post-operative analysis. This paper presents a novel method at the intersection of artificial intelligence and medicine, aiming to develop machine learning models with direct real-world applications in surgical contexts. We propose a multi-modal framework that leverages recent advancements in computer vision and large language models to generate comprehensive video summaries. % The approach is structured in three key stages. First, surgical videos are divided into clips, and visual features are extracted at the frame level using visual transformers. This step focuses on detecting tools, tissues, organs, and surgical actions. Second, the extracted features are transformed into frame-level captions via large language models. These are then combined with temporal features, captured using a ViViT-based encoder, to produce clip-level summaries that reflect the broader context of each video segment. Finally, the clip-level descriptions are aggregated into a full surgical report using a dedicated LLM tailored for the summarization task. % We evaluate our method on the CholecT50 dataset, using instrument and action annotations from 50 laparoscopic videos. The results show strong performance, achieving 96\% precision in tool detection and a BERT score of 0.74 for temporal context summarization. This work contributes to the advancement of AI-assisted tools for surgical reporting, offering a step toward more intelligent and reliable clinical documentation.

Enhancing Surgical Documentation through Multimodal Visual-Temporal Transformers and Generative AI

TL;DR

This work tackles automatic surgical video documentation by presenting a cohesive multimodal pipeline that fuses visual transformer-based feature extraction, frame- and clip-level captioning, and LLM-driven report synthesis. The method decomposes the task into object detection, caption generation, and narrative assembly, leveraging ViT, ViViT, DistilBERT, T5, and GPT-4 to produce structured surgical reports from laparoscopic videos. Through evaluation on the CholecT50 dataset, the approach achieves strong tool detection with calibrated probabilities, robust frame-captioning under noisy inputs, and coherent clip-level descriptions that feed into a final report—demonstrating the feasibility of end-to-end, explainable surgical documentation. The results underscore the practical impact of integrated multimodal reasoning for clinical documentation, training, and retrospective analysis, while outlining avenues for broader procedure coverage, longer temporal contexts, and multi-modal enhancements.

Abstract

The automatic summarization of surgical videos is essential for enhancing procedural documentation, supporting surgical training, and facilitating post-operative analysis. This paper presents a novel method at the intersection of artificial intelligence and medicine, aiming to develop machine learning models with direct real-world applications in surgical contexts. We propose a multi-modal framework that leverages recent advancements in computer vision and large language models to generate comprehensive video summaries. % The approach is structured in three key stages. First, surgical videos are divided into clips, and visual features are extracted at the frame level using visual transformers. This step focuses on detecting tools, tissues, organs, and surgical actions. Second, the extracted features are transformed into frame-level captions via large language models. These are then combined with temporal features, captured using a ViViT-based encoder, to produce clip-level summaries that reflect the broader context of each video segment. Finally, the clip-level descriptions are aggregated into a full surgical report using a dedicated LLM tailored for the summarization task. % We evaluate our method on the CholecT50 dataset, using instrument and action annotations from 50 laparoscopic videos. The results show strong performance, achieving 96\% precision in tool detection and a BERT score of 0.74 for temporal context summarization. This work contributes to the advancement of AI-assisted tools for surgical reporting, offering a step toward more intelligent and reliable clinical documentation.
Paper Structure (21 sections, 14 equations, 7 figures, 3 tables)

This paper contains 21 sections, 14 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed methodology
  4. Multi-Label Object Detection Using Vision Transformers
  5. Frame Caption Generation
  6. Clip-Level Caption Generation
  7. Comprehensive Surgical Report Synthesis
  8. Experimental Results
  9. Dataset and Preprocessing
  10. Setting up the experimental part
  11. Pre-trained Models
  12. Training Strategy
  13. Evaluation Metrics
  14. Object Detection
  15. Frame Caption Generation
  16. ...and 6 more sections

Figures (7)

  • Figure 1: Overview of different modalities in surgical video and applications.
  • Figure 2: Execution flow of the proposed methodology.
  • Figure 3: General scheme of the frame caption generation module.
  • Figure 4: Visual walkthrough of the proposed workflow applied to a sample surgical video clip.
  • Figure 5: Example of frame and clip caption generation from annotated surgical video.
  • ...and 2 more figures