Stability and Transparency in Mixed Reality Bilateral Human Teleoperation

TL;DR

This work analyzes stability and transparency in bilateral human teleoperation (HT) where an expert guides a novice wearing mixed-reality hardware during ultrasound guidance. By developing a mathematical HT model with time-delayed feedback, a hybrid transparency matrix, and multiple control architectures (2-channel, 3-channel, and model-mediated), the authors quantify performance under realistic delays and contact conditions using simulations and a real MR-haptic setup. Key findings show that small latencies (<200 ms) favor a three-channel approach with robust transparency, while large delays are best handled by model-mediated teleoperation with a local virtual model and impedance estimation. The results have practical implications for deploying low-resource MR-based teleguidance in remote communities, highlighting the need for accurate impedance modeling and incremental validation with broader user studies.

Abstract

Recent work introduced the concept of human teleoperation (HT), where the remote robot typically considered in conventional bilateral teleoperation is replaced by a novice person wearing a mixed reality head mounted display and tracking the motion of a virtual tool controlled by an expert. HT has advantages in cost, complexity, and patient acceptance for telemedicine in low-resource communities or remote locations. However, the stability, transparency, and performance of bilateral HT are unexplored. In this paper, we therefore develop a mathematical model and simulation of the HT system using test data. We then analyze various control architectures with this model and implement them with the HT system to find the achievable performance, investigate stability, and determine the most promising teleoperation scheme in the presence of time delays. We show that instability in HT, while not destructive or dangerous, makes the system impossible to use. However, stable and transparent teleoperation are possible with small time delays (<200 ms) through 3-channel teleoperation, or with large time delays through model-mediated teleoperation with local pose and force feedback for the novice.

Figures (12)

• Figure 1: Conceptual overview of the teleoperation system. The gray boxes of operator, follower, and patient have to be modeled while the haptic device, communications, and HoloLens 2 represent the part of the system we can directly control.
• Figure 2: A general model of human teleoperation. The operator (subscript $o$) interacts with a haptic device (subscript $h$) while the follower ($f$) interacts with the patient ($p$). The communication channel induces time delays of $T$ on the force and velocity which are transmitted bilaterally. The MR headset creates a visual control output potentially using all four channels of force and velocity, denoted $C_{HL}$. The follower responds to the MR input according to the follower transfer function, $G_f$. The controllers $C^i_j$ are on the follower or haptic device respectively with $i=f$ or $h$, and involve force or velocity respectively with $j=f$ or $v$.
• Figure 3: Models of the operator and haptic device (top), and the follower potentially in contact with the patient (bottom). The damping of the operator arm and haptic device together is represented by $b_o$. The black fixture attached to the follower mass is rigid and massless. The square end at position $x_v$ represents the virtual tool. The follower force $f_f=-k_px_f-b_p\dot{x}_f$ cancels the patient dynamics at all times. Thus, when the follower motion has converged (i.e. $b_f$ and $k_f$ are at equilibrium), the follower position $x_f$ should equal $x_v$.
• Figure 4: Interconnection of the derived models in the closed-loop teleoperation system. Different behavior is achieved by changing the feedback and feed-forward gains of the positions, velocities, and forces. The operator-side model is given in Equation \ref{['eqn:exptf']} while the follower side is in Equation \ref{['eqn:foltf']}.
• Figure 5: The derived model with parameters fitted to the measured data from previous tests black2023ijcarsblack2024tmrb. The MSE between the two is 0.67 mm.