Robust Immersive Bilateral Teleoperation of Beyond-Human-Scale Systems with Enhanced Transparency and Sense of Embodiment

TL;DR

The paper addresses robust bilateral teleoperation for beyond-human-scale hydraulic manipulators by integrating an immersive VR interface and a force-sensorless, VDC-based control architecture. It achieves high motion and force transparency while maintaining stability under unknown uncertainties and arbitrary time delays, leveraging a human–robot augmented dynamic model and SGUUB guarantees. Real-world experiments demonstrate precise tracking across large motion/force scaling (up to 1:13 and 1:1000 respectively) and delays up to 150 ms, plus a user study showing strong sense of embodiment (mean 76.4%) and usability among non-expert operators. This work offers a practical framework for immersive, embodied teleoperation of heavy-duty systems with potential for data-driven skill transfer and partial automation in industrial settings.

Abstract

In human-in-the-loop systems such as teleoperation, especially those involving heavy-duty manipulators, achieving high task performance requires both robust control and strong human engagement. This paper presents a bilateral teleoperation framework for beyond-human-scale robotic systems that enhances the transparency and the operator's sense of embodiment (SoE), specifically, the senses of agency and self-location, through an immersive virtual reality interface and distributed haptic feedback. To support this embodiment and establish high level of motion and force transparency, we develop a force-sensorless, robust control architecture that tackles input nonlinearities, master-surrogate asymmetries, unknown uncertainties, and arbitrary time delays. A human-robot augmented dynamic model is integrated into the control loop to enhance human-adaptability of the controller. Theoretical analysis confirms semi-global uniform ultimate boundedness of the closed-loop system, guaranteeing the robustness to the real-world uncertainties. Extensive real-world experiments demonstrate high accuracy tracking under up to 1:13 motion scaling and 1:1000 force scaling, showcasing the significance of the results. Additionally, the stability-transparency tradeoff for motion tracking and force reflection and tracking is established up to 150 ms of one-way fix and time-varying communication delays. The results of user study with 10 participants (9 male and 1 female) demonstrate that the system can imply a good level of SoE (76.4%), at the same time is very user friendly with no gender limitation. These results are significant given the scale and weight of the heavy-duty manipulators.

Figures (14)

• Figure 1: Scheme of the VR-based immersive bilateral teleoperation system with egocentric head tracking system. (a) Overview of the general teleoperation framework, featuring control units, communication channels, and a gaze-based interaction interface. By utilizing the eye-tracking capability of the VR headset, the interface enables the operator to switch between modes based on gaze direction. The side camera further enhances situational awareness, especially in cases where the operator encounters difficulties during manipulation. (b) Pan-tilt mechanism of the remote camera, synchronized with the human operator’s egocentric head movements to maintain intuitive visual feedback. (c) Visualization of the rendered and distributed scaled impedance of the surrogate onto the human operator’s arm. The utilized haptic exoskeleton enables the operator to perceive both the dynamics of the surrogate and its interactions with the remote environment.
• Figure 2: Proposed control method for robust and transparent force-reflected bilateral teleoperation. (a) Schematic of an immersive bilateral teleoperation system. (b) Graph-based representation of the proposed teleoperation control strategy. Open chains represent serial robotic manipulators; the "object" denotes the connective constraint linking two open chains; and the integrative open chain models the interconnected structure of the exoskeleton's serial links and the human arm. (c) Virtual decomposition of the master and surrogate systems. The integrative open chain highlights how the augmented human–robot model is seamlessly incorporated into the control framework. Details regarding the decomposition of the master robot and surrogate system can be found in hejrati2023physical and hejrati2023orchestrated, respectively.
• Figure 3: Decomposition of haptic exoskeleton using the VCP concept. a) Framing convention in VDC context, b) $i^{th}$ subsystem of decomposed model
• Figure 4: Scheme of 6-DoF HHM in presence of unknown environment
• Figure 5: Block diagram representation of dissimilar force-reflected bilateral teleoperation with arbitrary communication delay.

Theorems & Definitions (20)

• Definition 1
• Definition 2
• Lemma 1
• Lemma 2
• Lemma 3
• Definition 3
• Definition 4
• Remark 1
• Theorem 1
• Proof 1