ActiveUMI: Robotic Manipulation with Active Perception from Robot-Free Human Demonstrations

TL;DR

The paper tackles the data bottleneck in learning generalizable robot policies by introducing ActiveUMI, a portable data-collection framework that couples embodied human demonstrations with active perception. It deploys a VR teleoperation rig whose controllers map one-to-one to robot end-effectors, augmented by head-mounted camera control that the policy can actively maneuver, mitigating occlusions and enabling long-horizon tasks. Empirically, policies trained solely on ActiveUMI data achieve an average success of $70\%$ on six challenging bimanual tasks and generalize to novel objects/environments with $56\%$ average success, outperforming fixed-head and wrist-camera baselines by substantial margins. The work also demonstrates data-efficiency, showing that mixing a small amount of teleoperation data with in-the-wild ActiveUMI data yields near-saturation performance, highlighting a scalable path toward robust real-world robot policies that leverage learned active perception.

Abstract

We present ActiveUMI, a framework for a data collection system that transfers in-the-wild human demonstrations to robots capable of complex bimanual manipulation. ActiveUMI couples a portable VR teleoperation kit with sensorized controllers that mirror the robot's end-effectors, bridging human-robot kinematics via precise pose alignment. To ensure mobility and data quality, we introduce several key techniques, including immersive 3D model rendering, a self-contained wearable computer, and efficient calibration methods. ActiveUMI's defining feature is its capture of active, egocentric perception. By recording an operator's deliberate head movements via a head-mounted display, our system learns the crucial link between visual attention and manipulation. We evaluate ActiveUMI on six challenging bimanual tasks. Policies trained exclusively on ActiveUMI data achieve an average success rate of 70\% on in-distribution tasks and demonstrate strong generalization, retaining a 56\% success rate when tested on novel objects and in new environments. Our results demonstrate that portable data collection systems, when coupled with learned active perception, provide an effective and scalable pathway toward creating generalizable and highly capable real-world robot policies.

Figures (5)

• Figure 1: Overview of ActiveUMI Hardware. A VR headset with custom controllers designed to replicate the structure of the robot's grippers. A portable backpack that holds a battery and a PC for self-contained operation.
• Figure 2: Overview of ActiveUMI. The left side of the figure illustrates our data collection process and the detailed dataset configuration. The training data from in-the-wild data collected by ActiveUMI. The right side of the figure shows the model deployment and inference process.
• Figure 3: Immerse Data Collection. Our system provides the operator with critical visual feedback by rendering the robot's arms in the VR environment.
• Figure 4: Evaluated Tasks. We evaluated our approach on a diverse set of tasks, each requiring a different skill set: Block disassembly is a precision task where the robot must separate two small, interlocked blocks and then sort them into a box. Shirt folding is a deformable object manipulation task that demands accurate state recognition to correctly fold the cloth. Rope boxing is a long-horizon task where the robot must neatly place a long rope into a box. Toolbox cleaning is an articulated object manipulation task that requires the robot to close the lid. Bottle placing is a task designed to test the policy's robustness to large positional variations of the objects.
• Figure 5: Data Collection Comparison. (a)-(d) We utilize efficiency comparison among ActivateUMI, bare hand, and teleoperation in two tasks: rope boxing and shirt folding. ActivateUMI reaches an efficiency level between bare hand and teleoperation, and consistently outperforms teleoperation across both tasks.