ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity

TL;DR

This work addresses the need for fast, accurate, and scalable surgical simulation by introducing ORBIT-Surgical, a GPU-accelerated, photorealistic framework built on NVIDIA Isaac Sim. It provides 14 task benchmarks for dVRK and STAR, supports reinforcement and imitation learning, teleoperation, and synthetic data generation, and enables sim-to-real transfer to actual robotic hardware. Through extensive experiments, the paper demonstrates high-throughput RL/IL workflows, improved perception when combining simulated and real data, and successful real-world deployment on a dVRK system, highlighting the framework's potential to accelerate learning-based augmentation of surgical dexterity. The platform aims to lower the barrier to research in autonomous and augmented surgical robotics by delivering a unified, high-fidelity, and scalable simulation environment.

Abstract

Physics-based simulations have accelerated progress in robot learning for driving, manipulation, and locomotion. Yet, a fast, accurate, and robust surgical simulation environment remains a challenge. In this paper, we present ORBIT-Surgical, a physics-based surgical robot simulation framework with photorealistic rendering in NVIDIA Omniverse. We provide 14 benchmark surgical tasks for the da Vinci Research Kit (dVRK) and Smart Tissue Autonomous Robot (STAR) which represent common subtasks in surgical training. ORBIT-Surgical leverages GPU parallelization to train reinforcement learning and imitation learning algorithms to facilitate study of robot learning to augment human surgical skills. ORBIT-Surgical also facilitates realistic synthetic data generation for active perception tasks. We demonstrate ORBIT-Surgical sim-to-real transfer of learned policies onto a physical dVRK robot. Project website: orbit-surgical.github.io

Figures (7)

• Figure 1: (a) Physical da Vinci Research Kit (dVRK) platform, (b) Orbit-Surgical simulation.
• Figure 2: Orbit-Surgical Framework Architecture. The Orbit framework comprises of a simulation world, intelligent agents, and transfer of intelligence trained in simulation to the real world. The Orbit-Surgical simulation world includes realistic clones of surgical robots with joint articulation and low-level controllers. The object models include a rich suite of rigid and soft objects with precise collision properties. Sensory elements in the simulation imitate the real world sensors to provide training data. Peripheral I/O devices including the dVRK MTM are connected to the simulation environment to enable teleoperation of the simulated surgical robots. This setup facilitates the real-time capture of inputs from human experts, leveraging real-world demonstrations for policy learning in simulation.
• Figure 3: Reinforcement Learning. Success rate versus wall-clock time for RL policy training using the PPO algorithm on Reach task (above), and Rigid Object Manipulation task (below) evaluated for both Orbit-Surgical and LapGym across 10 trials for each saved checkpoint in simulation.
• Figure 4: Generalization of BC Policies in Simulation. We collected demonstrations using hand-scripted state trajectories with variations in initial states and goal states for Suture Needle Lift (Fig. \ref{['fig:fig1']}: Task3) in three different settings where only initial states are randomized (I), only goal states are randomized (G), and both initial and goal states are randomized (IG). Here, we compare the success rate of each trained policy across 10 trials in simulation. We note improved behavior for the policies trained on demonstrations with randomized initial and goal states.
• Figure 5: Synthetic Data Generation. Simulated images used as training data for needle segmentation (top). Real images of the workspace consisting of the needle are used for evaluation (bottom left). A model trained with both sim and real images is evaluated with a real dataset, resulting in masks shown on the bottom right.