Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation

Sampath Rapuri, Lalithkumar Seenivasan, Dominik Schneider, Roger Soberanis-Mukul, Yufan He, Hao Ding, Jiru Xu, Chenhao Yu, Chenyan Jing, Pengfei Guo, Daguang Xu, Mathias Unberath

Abstract

A surgical world model capable of generating realistic surgical action videos with precise control over tool-tissue interactions can address fundamental challenges in surgical AI and simulation -- from data scarcity and rare event synthesis to bridging the sim-to-real gap for surgical automation. However, current video generation methods, the very core of such surgical world models, require expensive annotations or complex structured intermediates as conditioning signals at inference, limiting their scalability. Other approaches exhibit limited temporal consistency across complex laparoscopic scenes and do not possess sufficient realism. We propose Surgical Action World (SAW) -- a step toward surgical action world modeling through video diffusion conditioned on four lightweight signals: language prompts encoding tool-action context, a reference surgical scene, tissue affordance mask, and 2D tool-tip trajectories. We design a conditional video diffusion approach that reformulates video-to-video diffusion into trajectory-conditioned surgical action synthesis. The backbone diffusion model is fine-tuned on a custom-curated dataset of 12,044 laparoscopic clips with lightweight spatiotemporal conditioning signals, leveraging a depth consistency loss to enforce geometric plausibility without requiring depth at inference. SAW achieves state-of-the-art temporal consistency (CD-FVD: 199.19 vs. 546.82) and strong visual quality on held-out test data. Furthermore, we demonstrate its downstream utility for (a) surgical AI, where augmenting rare actions with SAW-generated videos improves action recognition (clipping F1-score: 20.93% to 43.14%; cutting: 0.00% to 8.33%) on real test data, and (b) surgical simulation, where rendering tool-tissue interaction videos from simulator-derived trajectory points toward a visually faithful simulation engine.

SAW: Toward a Surgical Action World Model via Controllable and Scalable Video Generation

Abstract

A surgical world model capable of generating realistic surgical action videos with precise control over tool-tissue interactions can address fundamental challenges in surgical AI and simulation -- from data scarcity and rare event synthesis to bridging the sim-to-real gap for surgical automation. However, current video generation methods, the very core of such surgical world models, require expensive annotations or complex structured intermediates as conditioning signals at inference, limiting their scalability. Other approaches exhibit limited temporal consistency across complex laparoscopic scenes and do not possess sufficient realism. We propose Surgical Action World (SAW) -- a step toward surgical action world modeling through video diffusion conditioned on four lightweight signals: language prompts encoding tool-action context, a reference surgical scene, tissue affordance mask, and 2D tool-tip trajectories. We design a conditional video diffusion approach that reformulates video-to-video diffusion into trajectory-conditioned surgical action synthesis. The backbone diffusion model is fine-tuned on a custom-curated dataset of 12,044 laparoscopic clips with lightweight spatiotemporal conditioning signals, leveraging a depth consistency loss to enforce geometric plausibility without requiring depth at inference. SAW achieves state-of-the-art temporal consistency (CD-FVD: 199.19 vs. 546.82) and strong visual quality on held-out test data. Furthermore, we demonstrate its downstream utility for (a) surgical AI, where augmenting rare actions with SAW-generated videos improves action recognition (clipping F1-score: 20.93% to 43.14%; cutting: 0.00% to 8.33%) on real test data, and (b) surgical simulation, where rendering tool-tissue interaction videos from simulator-derived trajectory points toward a visually faithful simulation engine.
Paper Structure (12 sections, 3 figures, 3 tables)

This paper contains 12 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Surgical Action World (SAW) Model
  3. 2.1 Video Diffusion Model:
  4. 2.2 Lightweight Conditioning Signals:
  5. 2.3 Depth Consistency Loss:
  6. Experiment and Results
  7. Experimental Setup
  8. Results and Ablation Studies
  9. Downstream Applications
  10. 4.1 Synthetic Video Generation for Action Recognition:
  11. 4.2 Realistic Tool Tissue Interaction Engine for Surgical Simulation:
  12. Conclusion

Figures (3)

  • Figure 1: a) At inference, SAW leverages a user-inputted 2D tool trajectory, language prompt, tissue affordance, and a background surgical scene, enabling realistic and controllable video diffusion. b) During training, normalized depth and RGB videos are encoded using a frozen VAE, while the RGB video embedding is then flattened and noised into a sequence of noisy tokens. c) A depth consistency loss ($\mathcal{L}_{\text{DC}}$) is computed during training from the predicted masked depth video tokens solely using the denoised RGB video tokens.
  • Figure 2: Qualitative examples of generated surgical videos for (a) grasping and (b) dissecting actions. Red overlays indicate where a generated video lacks realism or becomes visually distorted in comparison to the ground truth video at the corresponding frame.
  • Figure 3: a) Simulator-derived tool segmentations, tissue affordance, tool tip trajectory. b) Simulator-derived tools are overlayed on a background surgical scene and used as conditional inputs to our video diffusion model, allowing us to model both tool kinematics and background tissue deformation.