HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation

TL;DR

HACMan tackles 6D non-prehensile manipulation by using an object-centric, temporally abstracted action representation that selects discrete contact locations from a object point cloud and continuous post-contact motions, all within a hybrid discrete-continuous RL framework. The method introduces per-point Actor and Critic Maps built on PointNet++ features, and represents the goal as per-point flow, enabling multimodal, goal-conditioned strategies and robust generalization to unseen objects. Simulation and real-world experiments show HACMan achieving strong performance on 6D pose alignment, with 89% success on unseen objects in simulation and 50% real zero-shot sim2real transfer, outperforming baselines by large margins. The results highlight the practicality of spatially-grounded action maps for complex, contact-rich non-prehensile skills and suggest promising directions for future 3D representations and broader non-prehensile manipulation tasks.

Abstract

Manipulating objects without grasping them is an essential component of human dexterity, referred to as non-prehensile manipulation. Non-prehensile manipulation may enable more complex interactions with the objects, but also presents challenges in reasoning about gripper-object interactions. In this work, we introduce Hybrid Actor-Critic Maps for Manipulation (HACMan), a reinforcement learning approach for 6D non-prehensile manipulation of objects using point cloud observations. HACMan proposes a temporally-abstracted and spatially-grounded object-centric action representation that consists of selecting a contact location from the object point cloud and a set of motion parameters describing how the robot will move after making contact. We modify an existing off-policy RL algorithm to learn in this hybrid discrete-continuous action representation. We evaluate HACMan on a 6D object pose alignment task in both simulation and in the real world. On the hardest version of our task, with randomized initial poses, randomized 6D goals, and diverse object categories, our policy demonstrates strong generalization to unseen object categories without a performance drop, achieving an 89% success rate on unseen objects in simulation and 50% success rate with zero-shot transfer in the real world. Compared to alternative action representations, HACMan achieves a success rate more than three times higher than the best baseline. With zero-shot sim2real transfer, our policy can successfully manipulate unseen objects in the real world for challenging non-planar goals, using dynamic and contact-rich non-prehensile skills. Videos can be found on the project website: https://hacman-2023.github.io.

Figures (23)

• Figure 1: We propose HACMan (Hybrid Actor-Critic Maps for Manipulation), which allows non-prehensile manipulation of unseen objects into arbitrary stable poses. With HACMan, the robot learns to push, tilt, and flip the object to reach the target pose, which is shown in the first column and in the top row with transparency. The policy allows for dynamic object motions with complex contact events in both simulation (top) and in the real world (bottom). The performance of the policy is best understood from the videos on the website: https://hacman-2023.github.io.
• Figure 2: Illustration of our action space.
• Figure 3: An overview of the proposed method. The point cloud observation includes the location of the points and point features. The goal is represented as per-point flow of the object points. The actor takes the observation as input and outputs an Actor Map of per-point motion parameters. The Actor Map is concatenated with the per-point critic features to generate the Critic Map of per-point Q-values. Finally, we choose the best contact location according to the highest value in the Critic Map and find the corresponding motion parameters in the Actor Map.
• Figure 4: Baselines and Ablations. Our approach outperforms the baselines and the ablations, with a larger margin for more challenging tasks on the right. Success rates for simple tasks - pushing a single object to an in-plane goal - are high for all methods, but only HACMan achieves high success rates for 6D alignment of diverse objects.
• Figure 5: Qualitative results for the object pose alignment task. HACMan shows complex non-prehensile behaviors that move the object to the goal pose (shown as the transparent object).