P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies

TL;DR

Prescriptive Point Priors for Policies (P3-PO) addresses generalization gaps in robot manipulation by using human-annotated semantic points that are propagated across demonstrations via diffusion-based correspondence and point tracking. The approach decouples perception from planning by feeding a point-based, depth-informed representation into a transformer policy, achieving strong spatial, novel-object, and distractor generalization. Across four real-world tasks in an xArm kitchen setup, P3-PO delivers substantial improvements over image-based baselines, including 43% overall, 58% on novel objects, and 80% under distractors. The work suggests that simple, human-guided priors can yield robust, data-efficient policies suitable for real-world deployment.

Abstract

Developing generalizable robot policies that can robustly handle varied environmental conditions and object instances remains a fundamental challenge in robot learning. While considerable efforts have focused on collecting large robot datasets and developing policy architectures to learn from such data, naively learning from visual inputs often results in brittle policies that fail to transfer beyond the training data. This work presents Prescriptive Point Priors for Policies or P3-PO, a novel framework that constructs a unique state representation of the environment leveraging recent advances in computer vision and robot learning to achieve improved out-of-distribution generalization for robot manipulation. This representation is obtained through two steps. First, a human annotator prescribes a set of semantically meaningful points on a single demonstration frame. These points are then propagated through the dataset using off-the-shelf vision models. The derived points serve as an input to state-of-the-art policy architectures for policy learning. Our experiments across four real-world tasks demonstrate an overall 43% absolute improvement over prior methods when evaluated in identical settings as training. Further, P3-PO exhibits 58% and 80% gains across tasks for new object instances and more cluttered environments respectively. Videos illustrating the robot's performance are best viewed at point-priors.github.io.

Figures (8)

• Figure 1: A human prescribes key points one time for one instance of an object and those points are transferable to all other instances of the same object.
• Figure 2: Overview of the Prescriptive Point Priors for Policies (P3-PO) framework. (a) A human annotator prescribes a set of semantically meaningful key points on a single demonstration frame, typically in under 5 seconds. Off-the-shelf computer vision models are then used to automatically propagate these key points throughout the entire dataset without further human input. (b) The derived key points are leveraged by a transformer policy to predict the action. (c) P3-PO enables learning policies with improved generalization capabilities, including spatial generalization (i.e. generalization to new locations), generalization to novel object instances, and robustness to background distractors. P3-PO combines the strengths of vision and policy prediction methods through simple yet effective human-prescribed semantic guidance.
• Figure 3: Results of the correspondence model when used on the pick mug and plate off rack tasks. On the left is the frame that is annotated by a human. On the right we show that semantic correspondence dift is able to identify the same points across a variety of instances of each object.
• Figure 4: Illustration of spatial variation used in our experiments.
• Figure 5: Illustration of objects used in our experiments. In each image, the left pile depicts the in-domain objects while on the right are novel objects used in our generalization experiments.