Dexterous Non-Prehensile Manipulation for Ungraspable Object via Extrinsic Dexterity

TL;DR

ExDex introduces a hierarchical framework for dexterous, non-prehensile manipulation of ungraspable objects by leveraging environmental affordances such as walls and table edges. A high-level planner selects optimal relocation targets and environmental contacts, while three low-level policies ($\pi_{push}$, $\pi_{wall}$, $\pi_{edge}$) learn non-prehensile manipulation through PPO in parallel simulation, with domain randomization and curriculum learning. The approach achieves zero-shot sim-to-real transfer using a teacher-student distillation pipeline and a digital-twin framework, enabling robust performance on real hardware with diverse objects, including deformables. Key contributions include the first exploration of extrinsic dexterity with dexterous hands in both simulation and real world, a novel three-policy hand–environment interaction strategy, and demonstrated generalization to unseen objects. The work advances practical extrinsic dexterity by combining strategic object relocation, dynamic environmental contacts, and precise manipulation control, with strong implications for real-world robotic manipulation tasks.

Abstract

Objects with large base areas become ungraspable when they exceed the end-effector's maximum aperture. Existing approaches address this limitation through extrinsic dexterity, which exploits environmental features for non-prehensile manipulation. While grippers have shown some success in this domain, dexterous hands offer superior flexibility and manipulation capabilities that enable richer environmental interactions, though they present greater control challenges. Here we present ExDex, a dexterous arm-hand system that leverages reinforcement learning to enable non-prehensile manipulation for grasping ungraspable objects. Our system learns two strategic manipulation sequences: relocating objects from table centers to edges for direct grasping, or to walls where extrinsic dexterity enables grasping through environmental interaction. We validate our approach through extensive experiments with dozens of diverse household objects, demonstrating both superior performance and generalization capabilities with novel objects. Furthermore, we successfully transfer the learned policies from simulation to a real-world robot system without additional training, further demonstrating its applicability in real-world scenarios. Project website: https://tangty11.github.io/ExDex/.

Figures (6)

• Figure 1: We present ExDex, a hierarchical framework that enables non-prehensile manipulation skills for ungraspable objects leveraging external environments. Our approach is demonstrated through two representative tasks: Wall, where objects are pushed against walls to facilitate manipulation. And Edge, where objects are repositioned to table edges allowing the hand to maneuver into optimal grasping poses. Through reinforcement learning in simulation, we successfully train these manipulation policies and achieve zero-shot transfer to real-world scenarios.
• Figure 2: Illustration of the ExDex System Design. (A) Training: Our system is trained in three stages. In Stage 1, we train a prediction model $\pi_\textrm{pre}$ through supervised learning that takes point cloud input and predicts the optimal target position $P_t$ for object repositioning. Stage 2 focuses on training three low-level skills via reinforcement learning: a pushing policy $\pi_\textrm{push}$ that repositions objects to target locations, and two policies $\pi_\textrm{wall}, \pi_\textrm{edge}$ that enable grasping of ungraspable objects from walls and table edges via extrinsic dexterity. In Stage 3, we jointly finetune these policies to ensure better transitions between consecutive skills. (B) Inference: During inference, our system first use the $\pi_\textrm{pre}$ to process the environmental point cloud to determine whether to execute the $\pi_\textrm{wall}$ or $\pi_\textrm{edge}$, while simultaneously predicting the corresponding target position $P_t$. The pushing policy $\pi_\textrm{push}$ then moves the object to this target position, followed by the selected extrinsic dexterity policy ($\pi_{\textrm{wall}}$ or $\pi_{\textrm{edge}}$) to complete the grasp.
• Figure 3: Overview of the environment setups. (a) Object sets used in simulation. Policies are firstly trained on the pretrain set, and then finetuned on the finetune set, and tested for zero-shot generalization on the unseen set. (b) Real-world test objects (top: wall-task objects, bottom: edge-task objects). (c) Workspace of the real-world, We use an Inspired Hand mounted on a UR5e robot, equipped with a RealSense D455 camera.
• Figure 4: Comparison of Arm-Only, Heuristic, and Ours for Wall task. (a) Arm-Only. (b) Heuristic. (c) Ours.
• Figure 5: Real-world experiment demonstrations. The snapshots show successful executions of our framework on various objects. (a) Wall tasks. (b) FrontEdge tasks. (c) LeftEdge tasks.