HRP: Human Affordances for Robotic Pre-Training

TL;DR

The paper tackles the robotics data bottleneck by introducing HRP, which mines human affordances from large-scale video data and distills them into pre-trained vision encoders. By predicting hand, object, and contact affordances, HRP enhances downstream robot learning when fine-tuned with behavior cloning across diverse tasks, robot morphologies, and camera views. HRP shows robust improvements over multiple strong baselines, outperforms label-based supervision, and generalizes to out-of-distribution distractors and alternative imitation-learning pipelines. The approach is designed to be data-efficient, modular, and readily open-sourced, with promising implications for scalable robotic pre-training.

Abstract

In order to *generalize* to various tasks in the wild, robotic agents will need a suitable representation (i.e., vision network) that enables the robot to predict optimal actions given high dimensional vision inputs. However, learning such a representation requires an extreme amount of diverse training data, which is prohibitively expensive to collect on a real robot. How can we overcome this problem? Instead of collecting more robot data, this paper proposes using internet-scale, human videos to extract "affordances," both at the environment and agent level, and distill them into a pre-trained representation. We present a simple framework for pre-training representations on hand, object, and contact "affordance labels" that highlight relevant objects in images and how to interact with them. These affordances are automatically extracted from human video data (with the help of off-the-shelf computer vision modules) and used to fine-tune existing representations. Our approach can efficiently fine-tune *any* existing representation, and results in models with stronger downstream robotic performance across the board. We experimentally demonstrate (using 3000+ robot trials) that this affordance pre-training scheme boosts performance by a minimum of 15% on 5 real-world tasks, which consider three diverse robot morphologies (including a dexterous hand). Unlike prior works in the space, these representations improve performance across 3 different camera views. Quantitatively, we find that our approach leads to higher levels of generalization in out-of-distribution settings. For code, weights, and data check: https://hrp-robot.github.io

Figures (9)

• Figure 1: Pre-trained representations offer a scalable solution to the robotics data bottleneck r3mrealmvpvc1, but existing methods fail to reliably improve over simple baselines like ImageNet dasari2023datasetsBurns2023WhatMakesPVR. Thus, we present HRP, a method that mines affordances (e.g., contact, hand pose, and object labels) from human videos and uses them to improve self-supervised visual encoders. Our best HRP representation consistently outperforms 6 SOTA baselines by $\geq \textbf{20\%}$ across 5 diverse tasks and 3 camera views.
• Figure 2: HRP fine-tunes a pre-trained encoder to predict three classes of human affordance labels via L2 regression. Specifically, the network must predict future contact points, human hand poses, and the target object given an input frame from the video stream. These affordance labels are mined autonomously from a human video dataset grauman2022ego4d using off-the-shelf vision detectors 100doh. HRP representations are then fine-tuned to solve downstream manipulation tasks via behavior cloning.
• Figure 3: We extract 3 affordances -- contact heatmaps, hand poses and active object bounding boxes -- from human videos.
• Figure 4: We present our policy training pipeline, which uses Behavior Cloning (BC) to train policy $\pi$, using optimal expert demonstrations. The image observation ($o_t$) is processed using our HRP representations resulting in a latent vector $z$. The policy uses $z$ to predict end-effector velocity actions (delta ee-pose/gripper), which are directly executed on the robot during test-time.
• Figure 5: Our experiments consider 5 unique manipulation tasks, ranging from classic block-stacking to a multi-stage toasting scenario. These tasks are implemented on 3 unique robot setups, including a high Degree-of-Freedom dexterous hand (right). The 3 camera views shown -- front, ego, and side views (for xArm/dexterous hand) -- are the same views ingested by the policy during test-time. Note that 3 of the tasks consider 2 unique camera views in order to test for robustness!