Towards Suturing World Models: Learning Predictive Models for Robotic Surgical Tasks

TL;DR

The paper addresses the lack of predictive world models for robotic suturing by learning high-fidelity, spatiotemporal representations of sub-stitch actions from expert-annotated video. It adapts two diffusion backbones, LTX-Video and HunyuanVideo, to the surgical domain, fine-tuning with both full parameter updates and LoRA on a dataset of 1,836 clips labeled as ideal or non-ideal. Key contributions include demonstrating the feasibility of diffusion-based suturing world models, comparing fine-tuning strategies, and reporting trade-offs between reconstruction fidelity and inference speed, with the best fidelity from HunyuanVideo at the cost of latency. The work lays groundwork for advanced simulators, skill assessment tools, and autonomous or semi-autonomous robotic systems in surgery, and provides a testable foundation through released models and a public project page.

Abstract

We introduce specialized diffusion-based generative models that capture the spatiotemporal dynamics of fine-grained robotic surgical sub-stitch actions through supervised learning on annotated laparoscopic surgery footage. The proposed models form a foundation for data-driven world models capable of simulating the biomechanical interactions and procedural dynamics of surgical suturing with high temporal fidelity. Annotating a dataset of $\sim2K$ clips extracted from simulation videos, we categorize surgical actions into fine-grained sub-stitch classes including ideal and non-ideal executions of needle positioning, targeting, driving, and withdrawal. We fine-tune two state-of-the-art video diffusion models, LTX-Video and HunyuanVideo, to generate high-fidelity surgical action sequences at $\ge$768x512 resolution and $\ge$49 frames. For training our models, we explore both Low-Rank Adaptation (LoRA) and full-model fine-tuning approaches. Our experimental results demonstrate that these world models can effectively capture the dynamics of suturing, potentially enabling improved training simulators, surgical skill assessment tools, and autonomous surgical systems. The models also display the capability to differentiate between ideal and non-ideal technique execution, providing a foundation for building surgical training and evaluation systems. We release our models for testing and as a foundation for future research. Project Page: https://mkturkcan.github.io/suturingmodels/

Figures (2)

• Figure 1: Workflow of the proposed approach. We prompt latent video diffusion models with text and (optionally) visual input. The models are trained with expert-annotated ideal and non-ideal demonstrations, allowing for the models to output either class of quality.
• Figure 2: Sample outputs of different models compared against a real-world needle driving clip from a backhand suturing task. (a) Real-world sample, (b) HunyuanVideo, (c) LTX Video LoRA, (d) LTX Video full training.