Hands-free teleoperation of a nearby manipulator through a virtual body-to-robot link

TL;DR

The paper addresses hands-free teleoperation of a nearby manipulator by linking a body part to the robot end-effector through a virtual pointer visualized in AR. It evaluates three control modes—Joystick, Body, and Dual—using a resolved-rate controller with an adjustable virtual pointer, showing that body-based control is intuitive and fast but more physically demanding, while the hybrid Dual Mode provides a practical compromise with predominantly body-driven control that can be augmented by a velocity-based joystick for larger tasks. Experimental results with fourteen participants reveal a strong natural bias toward body motions in Dual Mode, with significant reductions in variance across trials and potential applicability to assistive robotics, including head- or gaze-driven inputs for depth. The study highlights the tradeoffs between intuitive, small-range position control and broader, velocity-based control, and suggests future work on 6-DoF tasks and alternative attachment points for the virtual pointer to extend applicability in real-world assistive scenarios.

Abstract

This paper introduces an innovative control approach for teleoperating a robot in close proximity to a human operator, which could be useful to control robots embedded on wheelchairs. The method entails establishing a virtual connection between a specific body part and the robot's end-effector, visually displayed through an Augmented Reality (AR) headset. This linkage enables the transformation of body rotations into amplified effector translations, extending the robot's workspace beyond the capabilities of direct one-to-one mapping. Moreover, the linkage can be reconfigured using a joystick, resulting in a hybrid position/velocity control mode using the body/joystick motions respectively. After providing a comprehensive overview of the control methodology, we present the results of an experimental campaign designed to elucidate the advantages and drawbacks of our approach compared to the conventional joystick-based teleoperation method. The body-link control demonstrates slightly faster task completion and is naturally preferred over joystick velocity control, albeit being more physically demanding for tasks with a large range. The hybrid mode, where participants could simultaneously utilize both modes, emerges as a compromise, combining the intuitiveness of the body mode with the extensive task range of the velocity mode. Finally, we provide preliminary observations on potential assistive applications using head motions, especially for operators with limited range of motion in their bodies.

Figures (11)

• Figure 1: Representation of the set-up used during the experiments: the virtual end-effector ${\mathcal{F}_{{E_R^\star}}}$ (light blue sphere) is linked (light blue segment) to the user's thorax ${\mathcal{F}_{{E_H}}}$, whose movements are captured using Optitrack markers. This link is displayed using a virtual reality headset and can be reconfigured using a 3D joystick manipulated by the user's dominant hand. A robot, set nearby, with end-effector ${\mathcal{F}_{{E_R}}}$ (red sphere) is then servoed to follow the desired end-effector position. The link and spheres on the image are representations of what is seen through the AR headset. A frame ${\mathcal{F}_{{\bullet}}}$ is referenced by its three axes $X$, $Y$ and $Z$.
• Figure 2: A task example with 3 blue targets and 6 trajectories (numbered arrows). The task starts from the central point (red sphere), goes to one blue spherical target centered on the black sphere surface, and then back to center. Each target is completed when the robot end-effector $E_R$ has stayed for 1s less than 2cm away from the target center. Only 1 target appears at a time. Real trials consisted of 15 blue targets to reach.
• Figure 3: Distribution of the completion time per target for each participant (medians, 25th and 75th quartiles, and whiskers of width 1.96 the standard.
• Figure 4: TBD of the thorax per target in translation in cm
• Figure 5: TBD of the thorax per target in rotation in radians