A Digital Twin for Telesurgery under Intermittent Communication

TL;DR

This work tackles the problem of intermittent communication in telesurgery by introducing a digital twin framework that mirrors the da Vinci system and its environment. The approach calibrates a virtual model, overlays it via AR, and employs a replay-based recovery strategy during outages, with a baseline where teleoperation is disabled. Experimental results on a peg transfer task show a notable 23.6% reduction in completion time under replay (p < 0.005) and generally lower task load, demonstrating the practical potential of digital twins to stabilize remote surgical workflows and guide future outage-mitigation strategies.

Abstract

Telesurgery is an effective way to deliver service from expert surgeons to areas without immediate access to specialized resources. However, many of these areas, such as rural districts or battlefields, might be subject to different problems in communication, especially latency and intermittent periods of communication outage. This challenge motivates the use of a digital twin for the surgical system, where a simulation would mirror the robot hardware and surgical environment in the real world. The surgeon would then be able to interact with the digital twin during communication outage, followed by a recovery strategy on the real robot upon reestablishing communication. This paper builds the digital twin for the da Vinci surgical robot, with a buffering and replay strategy that reduces the mean task completion time by 23% when compared to the baseline, for a peg transfer task subject to intermittent communication outage. The relevant code can be found here: https://github.com/LCSR-CIIS/dvrk_digital_twin_teleoperation.

Figures (5)

• Figure 1: Our proposed digital twin framework for the da Vinci Research Kit to mitigate task disruptions due to communication losses in telesurgery applications. (a) During normal communication, the user sends control commands to both the real and digital twin robots. (b) During a communication outage, the user continues the task using the digital twin robot to minimize task disruption. (c) When communication is restored, the remote robot catches up with the commands of the digital twin.
• Figure 2: Transformations between different components of the digital twin system, separated into ones that are known and unknown (requiring calibrations).
• Figure 3: Software components of the digital twin system using AMBF simulator and ROS.
• Figure 4: Peg and board design. Pegs were designed with a countersink to encourage the user to grasp the peg from the same position every time. The close-up picture shows the tool holding the peg from the intended grasping position.
• Figure 5: Radar plot with the mean and standard deviation on each dimension of the NASA Task Load Index (TLX), across all participants, for the two different conditions. The dotted lines with shaded area indicate the standard deviation for each condition and dimension. The values closer to the center indicate lower task load.