Active Vision Might Be All You Need: Exploring Active Vision in Bimanual Robotic Manipulation

TL;DR

Addressing occlusion and limited FOV in imitation learning for dexterous bimanual manipulation, this paper introduces AV-ALOHA, a teleoperation system with a dedicated $7$-DoF AV arm controlled by a VR headset to actively adjust the camera viewpoint. It combines real-world and MuJoCo-based simulation experiments and evaluates an ACT imitation-learning policy across diverse tasks, with extensive ablations over seven camera configurations. The results show that human-guided active vision substantially improves performance on visibility-sensitive tasks, while static viewpoints remain advantageous for some simpler tasks, and using all cameras can hinder learning due to increased action complexity. The work provides open-source hardware, a simulation environment, and insights into view planning for robust, generalizable manipulation, highlighting both benefits and remaining challenges in decoupled vision and control, distribution shift, and data requirements.

Abstract

Imitation learning has demonstrated significant potential in performing high-precision manipulation tasks using visual feedback. However, it is common practice in imitation learning for cameras to be fixed in place, resulting in issues like occlusion and limited field of view. Furthermore, cameras are often placed in broad, general locations, without an effective viewpoint specific to the robot's task. In this work, we investigate the utility of active vision (AV) for imitation learning and manipulation, in which, in addition to the manipulation policy, the robot learns an AV policy from human demonstrations to dynamically change the robot's camera viewpoint to obtain better information about its environment and the given task. We introduce AV-ALOHA, a new bimanual teleoperation robot system with AV, an extension of the ALOHA 2 robot system, incorporating an additional 7-DoF robot arm that only carries a stereo camera and is solely tasked with finding the best viewpoint. This camera streams stereo video to an operator wearing a virtual reality (VR) headset, allowing the operator to control the camera pose using head and body movements. The system provides an immersive teleoperation experience, with bimanual first-person control, enabling the operator to dynamically explore and search the scene and simultaneously interact with the environment. We conduct imitation learning experiments of our system both in real-world and in simulation, across a variety of tasks that emphasize viewpoint planning. Our results demonstrate the effectiveness of human-guided AV for imitation learning, showing significant improvements over fixed cameras in tasks with limited visibility. Project website: https://soltanilara.github.io/av-aloha/

Figures (3)

• Figure 1: We introduce AV-ALOHA, a bimanual robot system with 7-DoF AV. In this system, a VR headset provides a live feed from the AV camera to the user. The movement of the VR headset controls the AV arm.
• Figure 2: Data collection and imitation learning pipeline with AV-ALOHA: The AV-ALOHA system enables intuitive data collection using a VR headset for AV and either VR controllers or leader arms for manipulation (left). This helps capture full body and head movements to teleoperate both our real and simulation system that record video from six different cameras (middle) and provide training data for our AV imitation learning policies (right).
• Figure 3: Robot Tasks: We experimented with five simulation and two real-world tasks. Some tasks encourage the robot to adjust its camera perspective.