VR Isle Academy: A VR Digital Twin Approach for Robotic Surgical Skill Development

TL;DR

The paper presents VR Isle Academy, a cost-effective VR digital twin for surgical robotic system (SRS) training designed to democratize access by being device-agnostic and portable. It integrates inside-out VR HMD, hand and feet tracking, and a real-time/offline analytics suite to monitor and score performance across eleven training scenarios. Built in Unity with the MAGES SDK, the system uses a six-DoF master-slave model and a Newton-Raphson IK solver to map user motions to robotic actions, including a mini-map for pedal feedback. Evaluations with untrained testers show performance improvements with practice, indicating the approach lowers the learning curve and training costs.

Abstract

Contemporary progress in the field of robotics, marked by improved efficiency and stability, has paved the way for the global adoption of surgical robotic systems (SRS). While these systems enhance surgeons' skills by offering a more accurate and less invasive approach to operations, they come at a considerable cost. Moreover, SRS components often involve heavy machinery, making the training process challenging due to limited access to such equipment. In this paper we introduce a cost-effective way to facilitate training for a simulator of a SRS via a portable, device-agnostic, ultra realistic simulation with hand tracking and feet tracking support. Error assessment is accessible in both real-time and offline, which enables the monitoring and tracking of users' performance. The VR application has been objectively evaluated by several untrained testers showcasing significant reduction in error metrics as the number of training sessions increases. This indicates that the proposed VR application denoted as VR Isle Academy operates efficiently, improving the robot - controlling skills of the testers in an intuitive and immersive way towards reducing the learning curve at minimal cost.

Figures (14)

• Figure 1: The da Vinci surgeon's console features distinct elements: the yellow triangle represents the machine's pedals, allowing users to enable/disable various robot functionalities. The light blue square denotes the masters, where the surgeon controls the robotic arms. Each master typically features two rings, into which the surgeon places their fingers in order to control the rotation and movement of the robotic arms. Lastly, the red circle indicates the output of the cameras. Figure from davinci_wikipedia
• Figure 2: The digital-twin robotic arm's end effector (left) and the master control of the machine (right). The master controls the end effector in the digital-twin VR Isle Academy. The red box highlights where the doctor's fingers should be placed in the master during the operation.
• Figure 3: Instructions for the Wrist Articulation 1 exercise. Each exercise includes instructional steps to guide the user, explaining their objectives and detailing what actions to take and what to avoid.
• Figure 4: Ring Tower Transfer 1 scenario. In this scenario, the user is tasked with removing a ring from a wire tower and placing it in a specific position on the sides of spherical objects. Applying excessive force with the forceps or moving too quickly can result in the tower detaching, leading to a failure.
• Figure 5: The Wrist Articulation 2 scenario in VR Isle Academy. The user is required to use the camera pedal to locate the correct ball. Subsequently, utilizing the clutch functionality, they must extend the robotic arms and, with precise wrist manipulations, touch the correct ball.