Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Assisting MoCap-Based Teleoperation of Robot Arm using Augmented Reality Visualisations

Qiushi Zhou, Antony Chacon, Jiahe Pan, Wafa Johal

TL;DR

The paper addresses the challenge of teleoperating a robot arm using motion capture by introducing augmented reality visualizations that render a virtual human-like arm to mediate the mapping between human motion and robot kinematics. Through two studies, it identifies that overlaying a human-like AR arm in the same orientation as the robot (HumanVertical) optimally supports learning and reduces perceived workload. The findings suggest AR overlays are most beneficial as a learning aid for novices, improving intuitive understanding of joint mappings without providing persistent burdens during operation. The approach demonstrates a practical MoCap-AR pipeline using FR3, HoloLens 2, and OptiTrack, with implications for more accessible, intuitive teleoperation in laboratory and industrial settings.

Abstract

Teleoperating a robot arm involves the human operator positioning the robot's end-effector or programming each joint. Whereas humans can control their own arms easily by integrating visual and proprioceptive feedback, it is challenging to control an external robot arm in the same way, due to its inconsistent orientation and appearance. We explore teleoperating a robot arm through motion-capture (MoCap) of the human operator's arm with the assistance of augmented reality (AR) visualisations. We investigate how AR helps teleoperation by visualising a virtual reference of the human arm alongside the robot arm to help users understand the movement mapping. We found that the AR overlay of a humanoid arm on the robot in the same orientation helped users learn the control. We discuss findings and future work on MoCap-based robot teleoperation.

Assisting MoCap-Based Teleoperation of Robot Arm using Augmented Reality Visualisations

TL;DR

The paper addresses the challenge of teleoperating a robot arm using motion capture by introducing augmented reality visualizations that render a virtual human-like arm to mediate the mapping between human motion and robot kinematics. Through two studies, it identifies that overlaying a human-like AR arm in the same orientation as the robot (HumanVertical) optimally supports learning and reduces perceived workload. The findings suggest AR overlays are most beneficial as a learning aid for novices, improving intuitive understanding of joint mappings without providing persistent burdens during operation. The approach demonstrates a practical MoCap-AR pipeline using FR3, HoloLens 2, and OptiTrack, with implications for more accessible, intuitive teleoperation in laboratory and industrial settings.

Abstract

Teleoperating a robot arm involves the human operator positioning the robot's end-effector or programming each joint. Whereas humans can control their own arms easily by integrating visual and proprioceptive feedback, it is challenging to control an external robot arm in the same way, due to its inconsistent orientation and appearance. We explore teleoperating a robot arm through motion-capture (MoCap) of the human operator's arm with the assistance of augmented reality (AR) visualisations. We investigate how AR helps teleoperation by visualising a virtual reference of the human arm alongside the robot arm to help users understand the movement mapping. We found that the AR overlay of a humanoid arm on the robot in the same orientation helped users learn the control. We discuss findings and future work on MoCap-based robot teleoperation.
Paper Structure (12 sections, 5 equations, 7 figures)

This paper contains 12 sections, 5 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Background
  3. Robot Teleoperation via Human Motion Mapping
  4. AR Visualisations and Robot Control
  5. MoCap-Based AR Teleoperation System
  6. Study 1: configuring the AR visualisation
  7. Results
  8. Discussion
  9. Study 2: Posture Control
  10. Results
  11. Discussion of Study 2
  12. Conclusion

Figures (7)

  • Figure 1: System architecture: a Linux machine running ROS 2 and the Franka Control Interface, a Windows machine running Unity and Motive, a Franka Research 3 robot, and a HoloLens 2 that renders through a remoting player.
  • Figure 2: Corresponding joint angles of human and robot arms.
  • Figure 3: AR Visualisation conditions and apparatus: (a) Participant situated away from the robot; (b) HumanHorizontal arm in AR next to the physical robot; (c) HumanVertical; (d) RobotHorizontal.
  • Figure 4: S2 postures: a) Elbow up, wrist up; b) Elbow up, wrist down; c) Elbow down, wrist up; d) Elbow down, wrist down.
  • Figure 5: Study 2: participant, condition No Arm and AR Arm. Blue and red sphere rendered on elbow and wrist respectively. Lighter blue and red denote targets to match posture.
  • ...and 2 more figures