Physical Human-Robot Interaction for Grasping in Augmented Reality via Rigid-Soft Robot Synergy

TL;DR

The paper addresses the challenge of teleoperating hybrid rigid–soft robots in unstructured environments by integrating an augmented reality interface with a physics-based simulation. It introduces a simulation-centered real-to-simulation parameter identification pipeline in MuJoCo to calibrate a cable-driven soft continuum arm, enabling accurate, real-time control. An AR-based teleoperation framework then uses dual AR joysticks to command both the rigid arm and the soft tip, with a virtual preview to reduce real-world trial-and-error. Experimental results show significant reduction in internal shape error and demonstrate successful reaching, following, and grasping tasks, highlighting the approach's potential for safer and more versatile manipulation in complex environments.

Abstract

Hybrid rigid-soft robots combine the precision of rigid manipulators with the compliance and adaptability of soft arms, offering a promising approach for versatile grasping in unstructured environments. However, coordinating hybrid robots remains challenging, due to difficulties in modeling, perception, and cross-domain kinematics. In this work, we present a novel augmented reality (AR)-based physical human-robot interaction framework that enables direct teleoperation of a hybrid rigid-soft robot for simple reaching and grasping tasks. Using an AR headset, users can interact with a simulated model of the robotic system integrated into a general-purpose physics engine, which is superimposed on the real system, allowing simulated execution prior to real-world deployment. To ensure consistent behavior between the virtual and physical robots, we introduce a real-to-simulation parameter identification pipeline that leverages the inherent geometric properties of the soft robot, enabling accurate modeling of its static and dynamic behavior as well as the control system's response.

Figures (6)

• Figure 1: Schematic diagram of the proposed AR interaction method.
• Figure 2: (a) Restricted workspace of the rigid robot. (b) Reachability map of the spiral-shaped soft arm with its backbone aligned with the gravity direction. (c) Reachability maps of our soft arm at $0^\circ$, $60^\circ$, and $120^\circ$ relative to the gravity direction. (d) Visualization of the restricted reachability map overlaid on the AR robot model.
• Figure 3: (a). Experimental setup of our soft arm and the motion capture system with six infrared cameras. (b). Static tilting experiment to identify bending stiffness for each section, corresponds to Table \ref{['table1']} Exp.1. (c) Curling and uncurling in different actuation statuses to identify damping properties for the robot and the actuator properties for the control system, corresponds to Table \ref{['table1']} Exp.2 and 3.
• Figure 4: Additional details are provided in the supplementary video. (a) Adjustment of the ray length projected from the joystick; (b) Tilting the joystick to change the orientation of the rigid robot for alignment with the joystick. (c) Control of the soft robot’s curling and uncurling motions by specifying the end-effector position using the ray. (d) Control of the rigid robot’s end-effector position using the ray. (e) Overview of the joystick functions.
• Figure 5: (a) Uncurling motion of the physical robot with motion-capture markers illustrating the trajectory over time. (b) Simulated uncurling motion after parameter identification. (c) Simulated uncurling motion using initial, unoptimized parameters. (d) Positional error between the optimised and unoptimized models. (e) Velocity error between the optimized and unoptimized models.