Vision-Based Manipulators Need to Also See from Their Hands

TL;DR

This work analyzes how visual perspective influences learning and generalization in vision-based robotic manipulation, showing that hand-centric observations ($O_h$) improve training efficiency and out-of-distribution generalization when observability is adequate. When $O_h$ is insufficient, combining hand-centric and third-person views with a variational information bottleneck on the third-person stream yields state-of-the-art generalization across multiple tasks and learning algorithms. The approach is validated in simulation and on real robots, including six Meta-World tasks and a sponge-grasping real-robot setup, and demonstrated to reduce out-of-distribution failure rates by substantial margins. The findings provide broadly applicable guidelines for designing end-to-end perception-action loops in manipulation, with simple, effective regularization that mitigates overfitting to training conditions. Overall, the paper argues for strategic camera placement and representation regularization to enhance robust visuomotor learning.

Abstract

We study how the choice of visual perspective affects learning and generalization in the context of physical manipulation from raw sensor observations. Compared with the more commonly used global third-person perspective, a hand-centric (eye-in-hand) perspective affords reduced observability, but we find that it consistently improves training efficiency and out-of-distribution generalization. These benefits hold across a variety of learning algorithms, experimental settings, and distribution shifts, and for both simulated and real robot apparatuses. However, this is only the case when hand-centric observability is sufficient; otherwise, including a third-person perspective is necessary for learning, but also harms out-of-distribution generalization. To mitigate this, we propose to regularize the third-person information stream via a variational information bottleneck. On six representative manipulation tasks with varying hand-centric observability adapted from the Meta-World benchmark, this results in a state-of-the-art reinforcement learning agent operating from both perspectives improving its out-of-distribution generalization on every task. While some practitioners have long put cameras in the hands of robots, our work systematically analyzes the benefits of doing so and provides simple and broadly applicable insights for improving end-to-end learned vision-based robotic manipulation.

Figures (20)

• Figure 1: Illustration suggesting the role that visual perspective can play in facilitating the acquisition of symmetries with respect to certain transformations on the world state $\mathbf{s}$. $T_0$: planar translation of the end-effector and cube. $T_1$: vertical translation of the table surface, end-effector, and cube. $T_2$: addition of distractor objects. $O_3$: third-person perspective. $O_h$: hand-centric perspective.
• Figure 2: DAgger and DrQ results for cube grasping. The first, second, and third rows respectively contain results for the table height (shifted by $z_\text{shift}$), distractor objects, and table textures experiment variants. See Appendix \ref{['app:cube_environment_details']} for visualizations of the train and test distributions for each experiment variant. Compared to the third-person perspective (dashed lines), the hand-centric perspective (solid lines) leads to better out-of-distribution generalization performance across all three distribution shifts for both DAgger and DrQ. For DrQ, we also see appreciable improvements in sample efficiency when using the hand-centric perspective. Shaded regions indicate the standard error of the mean over three random seeds.
• Figure 3: DAC results for cube grasping. Left: base variant (initial object and end-effector position randomization) with no distribution shift between demo collection and training. Center: base variant with table height shift between collection of 25 demos and training. Right: base variant plus three distractor objects with no distribution shift between demo collection and training. Across the three experiment variants, the hand-centric perspective enables the agent to generalize in- and out-of-distribution more efficiently and effectively. Shaded regions indicate the standard error of the mean over five random seeds.
• Figure 4: Sample observations from $O_3$ (left) and $O_h$ (right) in our real robot apparatus.
• Figure 5: The Meta-World tasks used in the experiments in Section \ref{['sec:view_1_and_view_3']}. The top row contains third-person observations $\mathbf{o}_3$, and the bottom row contains corresponding hand-centric observations $\mathbf{o}_h$. Initial object positions are randomized. The last two tasks, reach-hard and peg-insert-side-hard, are custom-made; there, the green goal and the green peg are randomly initialized either to the left or to the right of the gripper with equal probability, and they are not initially visible to the hand-centric perspective. Because of the severely limited hand-centric observability, the third-person perspective is crucial for learning to direct the gripper to the correct location. This is especially the case for the reach-hard task, which we modified to prohibit any vertical end-effector movement. See Section \ref{['app:metaworld_individual_task_descriptions']} in the Appendix for more details about each task.