Puppeteer Your Robot: Augmented Reality Leader-Follower Teleoperation

TL;DR

High-quality demonstrations are essential for learning complex manipulation, and intuitive teleoperation is needed without duplicating hardware. The paper proposes an augmented reality leader-follower Puppeteer Your Robot system that uses a virtual leader in AR to guide a real follower robot, with inverse kinematics mapping and ROS-TCP communication, plus visual latency cues via a transparent overlay. Key contributions include the novel AR-based puppeteering framework for the Franka Panda and an empirical pilot study with n=10 participants demonstrating favorable usability and task performance in block stacking and rice scooping. The work highlights significant practical impact by reducing hardware requirements while maintaining intuitive control and safety, enabling more scalable collection of high-quality demonstrations in real-world settings.

Abstract

High-quality demonstrations are necessary when learning complex and challenging manipulation tasks. In this work, we introduce an approach to puppeteer a robot by controlling a virtual robot in an augmented reality setting. Our system allows for retaining the advantages of being intuitive from a physical leader-follower side while avoiding the unnecessary use of expensive physical setup. In addition, the user is endowed with additional information using augmented reality. We validate our system with a pilot study n=10 on a block stacking and rice scooping tasks where the majority rates the system favorably. Oculus App and corresponding ROS code are available on the project website: https://ar-puppeteer.github.io/

Figures (7)

• Figure 1: Our method uses an augmented reality (AR) approach to enable puppeteering without duplication of the real hardware. A virtual robot (right) is used as leader and the real robot (left) acts as follower.
• Figure 2: An overview of our AR puppeteer system. The Oculus projects the virtual robot into the line of sight of the user, who then uses the controller to grasp the virtual robot at the end-effector and subsequently move it around. The resulting virtual joint positions are sent to a ROS Node and translated into suitable control signals for the real robot. Furthermore, the real robot's current joint position is relayed back to the Occulus over the same ROS node and enables the visualization between the wanted and current position of the robotic arm by visualizing the current position as a transparent green version of the robot. The user sees the composited augmented reality view (left) composed of the virtual robot, and the real robot.
• Figure 3: Setup of AR puppeteering in steps: a) Greeting screen, b) set ROS IP and Port to connect to, c) Setup completed screen, d) Position the controller to spawn the virtual robot, e) Virtual robot spawned, f) Pupeeterring engaged.
• Figure 4: Overview of how the AR Puppeteer control is realized. The virtual joint positions are obtained via Inverse kinematics given the virtual end-effector pose. These joint positions are then published onto a ROS topic where a PD controller realizes them on the real robot, the kinematics of the robot then results in the real end-effector following the virtual one.
• Figure 5: Visulisation of the state of the follower robot overlayed as a transparent robot to indicate the alignment between leader and follower in an intuitive way. Left: a green transparent robot indicates the delay between the leader and follower. Right: a red transparent robot indicates that the follower diverged from the leader and that realignment is needed.