Where to Touch, How to Contact: Hierarchical RL-MPC Framework for Geometry-Aware Long-Horizon Dexterous Manipulation

TL;DR

This work introduces a hierarchical RL–MPC framework that decomposes dexterous, contact-rich manipulation into high-level geometric–kinematic planning and low-level contact dynamics control. The novel contact-intention interface specifies where to touch and the post-contact object subgoal, enabling a high-level policy to guide a fast, physics-based MPC (ComFree-MPC) that handles local contact modes at ~100 Hz. Key innovations include an object-centric tri-component observation, a dual-branch RL policy for contact-point selection and subgoal weighting, and a complementarity-free MPC that robustly optimizes contact strategies. The approach achieves near-100% success on geometry-generalized pushing and 3D reorientation tasks with substantially less data, exhibits strong robustness to environment variations, and transfers zero-shot from simulation to real robots, indicating strong practical impact for geometry-aware long-horizon manipulation.

Abstract

A key challenge in contact-rich dexterous manipulation is the need to jointly reason over geometry, kinematic constraints, and intricate, nonsmooth contact dynamics. End-to-end visuomotor policies bypass this structure, but often require large amounts of data, transfer poorly from simulation to reality, and generalize weakly across tasks/embodiments. We address those limitations by leveraging a simple insight: dexterous manipulation is inherently hierarchical - at a high level, a robot decides where to touch (geometry) and move the object (kinematics); at a low level it determines how to realize that plan through contact dynamics. Building on this insight, we propose a hierarchical RL--MPC framework in which a high-level reinforcement learning (RL) policy predicts a contact intention, a novel object-centric interface that specifies (i) an object-surface contact location and (ii) a post-contact object-level subgoal pose. Conditioned on this contact intention, a low-level contact-implicit model predictive control (MPC) optimizes local contact modes and replans with contact dynamics to generate robot actions that robustly drive the object toward each subgoal. We evaluate the framework on non-prehensile tasks, including geometry-generalized pushing and object 3D reorientation. It achieves near-100% success with substantially reduced data (10x less than end-to-end baselines), highly robust performance, and zero-shot sim-to-real transfer.

Figures (12)

• Figure 1: Overview of our hierarchical RL--MPC framework for geometry-aware, long-horizon non-prehensile manipulation. A shared high-level RL policy performs geometric–kinematic reasoning and predicts a contact intention (object-surface contact locations and a post-contact object-level subgoal) across diverse non-convex geometries and tasks. Conditioned on it, a lower-level contact-implicit MPC performs contact dynamics reasoning locally, computing robot actions that realize the contact intention.
• Figure 2: Illustration of the RL-MPC hierarchical framework
• Figure 3: Illustration of the geometric, target and collision components in RL observation space. The green mesh stands for current object pose, and the red mesh stands for the manipulation target.
• Figure 4: The dual-branch architecture design for the higher-level RL policy that predicts contact intention.
• Figure 5: Two non-prehensile manipulation tasks. Left: Geometry-generalized pushing. Right: object 3D reorientation.