Learning to Look Around: Enhancing Teleoperation and Learning with a Human-like Actuated Neck

TL;DR

The benefits of natural perception across seven challenging teleoperation tasks are demonstrated, showing how the actuated neck enhances the scope and efficiency of remote operation and its role in training autonomous policies through imitation learning is investigated.

Abstract

We introduce a teleoperation system that integrates a 5 DOF actuated neck, designed to replicate natural human head movements and perception. By enabling behaviors like peeking or tilting, the system provides operators with a more intuitive and comprehensive view of the environment, improving task performance, reducing cognitive load, and facilitating complex whole-body manipulation. We demonstrate the benefits of natural perception across seven challenging teleoperation tasks, showing how the actuated neck enhances the scope and efficiency of remote operation. Furthermore, we investigate its role in training autonomous policies through imitation learning. In three distinct tasks, the actuated neck supports better spatial awareness, reduces distribution shift, and enables adaptive task-specific adjustments compared to a static wide-angle camera.

Figures (6)

• Figure 1: A teleoperation system featuring an actuated neck and dexterous arms, enabling human-like manipulation in complex environments. The robot's 5-DOF neck mimics natural head movements, allowing behaviors like "peeking" obstacles to locate objects. By incorporating human-like perception, the system broadens task capabilities and enhances efficiency, paving the way for advanced remote teleoperation.
• Figure 2: Overview of the teleoperated robotic system featuring a mobile base, 5-DOF actuated neck interbotix, dual UR5e arms ur5, and dexterous grippers robotiq_gripperpsyonic. The diagram details key dimensions, capabilities, and teleoperator setup, which includes Vision Pro for head tracking and hand tracking devices. The system is designed to provide human-like movement and precise control for remote teleoperation tasks, with a total of 21 degrees of freedom (14 for arms with parallel-jaw grippers, 5 for neck, and 2 for base) and additional 6 DoF with a Psyonic Ability dexterous hand.
• Figure 3: Demonstrating different head movements during teleoperation, mapped directly to the robot’s actuated neck: (Top left) Neutral, (Top right) Sideways, (Bottom left) Peeking, (Bottom right) Slanting. The robot’s 5-DOF neck enables natural, human-like adjustments, enhancing perception and control in complex environments.
• Figure 4: Teleoperated Tasks: The teleoperated robot performing various manipulation tasks, including setting up a dinner table (left), retrieving items from a refrigerator (middle), and managing workspace clutter (right). The 5-DOF actuated neck and dexterous hands enable precise control and adaptation to diverse environments, facilitating human-like interaction with household objects during teleoperation.
• Figure 5: Autonomous Tasks: A single policy is trained on three merged tasks across varying workspace heights and positions. (Left) The robot "peeks" and picks up a cup, placing it on a top shelf. (Middle) The robot picks an object close to its body and places it on a table. (Right) The robot retrieves an item from under a coffee table and deposits it inside a package. The actuated neck adjusts dynamically based on observations, allowing the robot to identify and complete tasks without specific task labeling.