SurgWorld: Learning Surgical Robot Policies from Videos via World Modeling

TL;DR

SurgWorld tackles data scarcity in autonomous surgical robotics by coupling a large-scale surgical world model with a carefully annotated SATA dataset and an inverse dynamics model to generate pseudo-kinematics from synthetic videos. The approach enables scalable, data-efficient policy learning for surgical robots by training VLA models on a mix of real demonstrations and synthetic, text-grounded video data. Key contributions include the SATA dataset, the first surgical world model fine-tuned on surgical text annotations, and the synthesis of video–action pairs to boost policy learning, demonstrated on needle pickup and handover tasks with improved metrics and expert-rated realism. This work offers a practical pathway to scalable autonomous surgical skill acquisition while highlighting limitations in domain transfer to unseen embodiments and the noisiness of pseudo-kinematics, guiding future enhancements in dataset breadth and IDM fidelity.

Abstract

Data scarcity remains a fundamental barrier to achieving fully autonomous surgical robots. While large scale vision language action (VLA) models have shown impressive generalization in household and industrial manipulation by leveraging paired video action data from diverse domains, surgical robotics suffers from the paucity of datasets that include both visual observations and accurate robot kinematics. In contrast, vast corpora of surgical videos exist, but they lack corresponding action labels, preventing direct application of imitation learning or VLA training. In this work, we aim to alleviate this problem by learning policy models from SurgWorld, a world model designed for surgical physical AI. We curated the Surgical Action Text Alignment (SATA) dataset with detailed action description specifically for surgical robots. Then we built SurgeWorld based on the most advanced physical AI world model and SATA. It's able to generate diverse, generalizable and realistic surgery videos. We are also the first to use an inverse dynamics model to infer pseudokinematics from synthetic surgical videos, producing synthetic paired video action data. We demonstrate that a surgical VLA policy trained with these augmented data significantly outperforms models trained only on real demonstrations on a real surgical robot platform. Our approach offers a scalable path toward autonomous surgical skill acquisition by leveraging the abundance of unlabeled surgical video and generative world modeling, thus opening the door to generalizable and data efficient surgical robot policies.

Figures (14)

• Figure 1: We curate SATA dataset with surgical videos and detailed text annotations for physical AI. A powerful world model (SurgWorld) is built using Cosmos2.5 nvidia2025worldsimulationvideofoundation and SATA, which is able to generate high quality, generalizable videos for surgical robots. We are also the first to illustrate the efficacy of surgical world modeling for autonomous surgical robots.
• Figure 2: The overall workflow. The SurgWorld model is first pretrained with large scale surgical videos with text annotations, based on Cosmos 2.5 nvidia2025worldsimulationvideofoundation. For downstream task with specific robot type and task, we finetune SurgWorld and train the inverse dynamic model (IDM) for the specific embodiment. In step 3 we generate synthetic video rollouts from SurgWorld and get pseudo kinematics from the IDM. We use both real data and synthetic data to train the surgical VLA model.
• Figure 3: The model architecture for inverse dynamic models (IDM) and the vision language action foundation model (GR00T N1.5). They share similar architectures but IDM does not use text prompt nor robot state.
• Figure 4: Qualitative comparison of three variants of Cosmos-Predict2.5 nvidia2025worldsimulationvideofoundation on the SATA dataset. Red arrows highlight incorrect surgical tools or actions in the generated frames.
• Figure 5: New behavior generalization via strong text–video alignment. Given the same conditioning frame, our surgical world model generates distinct video rollouts corresponding to four task prompts: (1) one-time needle handover, (2) two-time needle handover, (3) three-time needle handover, and (4) needle puncture.