PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations

TL;DR

PartManip tackles cross-category generalization in object manipulation by combining a part-aware, canonicalized state-based expert with a vision-based student trained via DAgger and reinforced by a powerful 3D Sparse UNet backbone. The approach uses part pose-aware dense rewards and domain-adversarial learning to promote cross-category feature invariance, achieving strong performance on unseen object categories and translating to real-world manipulation through a digital twin setup. Key contributions include the first large-scale part-based cross-category benchmark, a novel part-canonicalization strategy for RL, a robust state-to-vision distillation pipeline, and demonstrated sim-to-real transfer. The work advances generalizable manipulation by integrating structured part representations, dense rewards, expressive vision backbones, and domain generalization techniques, with practical implications for robust robotic interaction in diverse environments.

Abstract

Learning a generalizable object manipulation policy is vital for an embodied agent to work in complex real-world scenes. Parts, as the shared components in different object categories, have the potential to increase the generalization ability of the manipulation policy and achieve cross-category object manipulation. In this work, we build the first large-scale, part-based cross-category object manipulation benchmark, PartManip, which is composed of 11 object categories, 494 objects, and 1432 tasks in 6 task classes. Compared to previous work, our benchmark is also more diverse and realistic, i.e., having more objects and using sparse-view point cloud as input without oracle information like part segmentation. To tackle the difficulties of vision-based policy learning, we first train a state-based expert with our proposed part-based canonicalization and part-aware rewards, and then distill the knowledge to a vision-based student. We also find an expressive backbone is essential to overcome the large diversity of different objects. For cross-category generalization, we introduce domain adversarial learning for domain-invariant feature extraction. Extensive experiments in simulation show that our learned policy can outperform other methods by a large margin, especially on unseen object categories. We also demonstrate our method can successfully manipulate novel objects in the real world.

Figures (5)

• Figure 1: Overview. We introduce a large-scale cross-category part manipulation benchmark PartManip with diverse object datasets, realistic settings, and rich annotations. We propose a generalizable vision-based policy learning strategy and boost the performance of part-based object manipulation by a large margin, which can generalize to unseen object categories and novel objects in the real world.
• Figure 2: Object Assets Visualization. The object geometry and appearance are very different, especially in different object categories, which presents a great challenge for our PartManip benchmark.
• Figure 3: Our Pipeline. We first train state-based expert policy using our proposed canonicalization to the part coordinate frame and the part-aware reward. We then use the learned expert to collect demonstrations for pre-training the vision-based policy by behavior cloning. After pre-training, we train the vision-based policy to imitate the state-based expert policy using DAgger. We also introduce several point cloud augmentation techniques to boost the generalization ability. For the vision backbone, we introduce 3D Sparse-UNet which has a large expression capability. Furthermore, we introduced an extra domain adversarial learning module for better cross-category generalization.
• Figure 4: Real World Experiment.
• Figure 5: Failure Cases