Improving the realism of robotic surgery simulation through injection of learning-based estimated errors

TL;DR

This work tackles the gap between simulated and real robotic surgery systems by learning to inject realistic error patterns into a simulator. It decomposes errors into controller-related and mechanical components, trains two neural networks to predict joint-space offsets from real robot data, and injects these offsets into the simulated robot to reproduce the real robot's motion distributions. The approach yields substantial reductions in mean position and orientation discrepancies between real and simulated trajectories ($E_T$ dropping from $5.0\pm2.0$ mm to $1.3\pm0.6$ mm and $E_R$ from $3.6\pm1.4$ deg to $1.7\pm0.7$ deg), enhancing realism for autonomous-surgery algorithm development. By focusing on statistical similarity rather than exact compensation, this method facilitates robust learning and testing of automation tasks in scenarios with inherently imperfect robots.

Abstract

The development of algorithms for automation of subtasks during robotic surgery can be accelerated by the availability of realistic simulation environments. In this work, we focus on one aspect of the realism of a surgical simulator, which is the positional accuracy of the robot. In current simulators, robots have perfect or near-perfect accuracy, which is not representative of their physical counterparts. We therefore propose a pair of neural networks, trained by data collected from a physical robot, to estimate both the controller error and the kinematic and non-kinematic error. These error estimates are then injected within the simulator to produce a simulated robot that has the characteristic performance of the physical robot. In this scenario, we believe it is sufficient for the estimated error used in the simulation to have a statistically similar distribution to the actual error of the physical robot. This is less stringent, and therefore more tenable, than the requirement for error compensation of a physical robot, where the estimated error should equal the actual error. Our results demonstrate that error injection reduces the mean position and orientation differences between the simulated and physical robots from 5.0 mm / 3.6 deg to 1.3 mm / 1.7 deg, respectively, which represents reductions by factors of 3.8 and 2.1.

Figures (6)

• Figure 1: Proposed system to emulate the real-robot motion patterns using a simulated dVRK.
• Figure 2: Transformation diagram that indicates how to calculate the robot's actual pose ($A_1$) with an optical tracker sensor.
• Figure 3: Hyperparameter optimization experiment for $NN_1$. The legend represents: {Number of neurons in hidden layers}-{Type of input features}. Three input features were considered: Only current (OC), Current+previous (CP), and Current+previous encoded (CPE) (see subsection \ref{['subsect-neural-net-modelling']}). It is observed that adding previous measurements is necessary to reduce the validation loss.
• Figure 4: Qualitative results for the neural networks. Left plot shows predictions for the controller error while the right plot shows predictions for kinematic/non-kinematic errors. Ground-truth offsets for each of the joints of the dVRK are shown in blue, while predicted offsets are shown in orange. Orange lines following closely the blue lines indicate that the neural networks are correctly modeling the robot errors. Notice that the dVRK has 5 rotational and 1 translational joints and therefore not all the offsets have the same units.
• Figure 5: Experiment setup. The marker was rigidly attached to the tip of the tool, and it was tracked by the optical tracker.