Robust Model-Based In-Hand Manipulation with Integrated Real-Time Motion-Contact Planning and Tracking

TL;DR

This work tackles robust dexterous in-hand manipulation under contact-rich dynamics and disturbances by introducing a hierarchical, model-based framework that couples real-time motion-contact planning with tactile-feedback tracking. The high-level planner uses a contact-implicit MPC based on a differentiable CQDC implicit dynamics model solved with Differential Dynamic Programming (DDP), while the low-level tracker uses an MPC-based hybrid force-motion controller that accounts for multi-contact coupling. Key contributions include smoothing of contact dynamics with a barrier parameter, a gradient computation strategy including numerical differentiation for collision-sensitive terms, warm-starting to ensure real-time performance, and an automated weighting scheme for force and motion tracking. Extensive simulation and real-world experiments demonstrate improved accuracy, robustness to modeling errors and noise, and real-time capability across diverse tasks and object geometries, highlighting the method’s potential for generalizable, training-free dexterous manipulation.

Abstract

Robotic dexterous in-hand manipulation, where multiple fingers dynamically make and break contact, represents a step toward human-like dexterity in real-world robotic applications. Unlike learning-based approaches that rely on large-scale training or extensive data collection for each specific task, model-based methods offer an efficient alternative. Their online computing nature allows for ready application to new tasks without extensive retraining. However, due to the complexity of physical contacts, existing model-based methods encounter challenges in efficient online planning and handling modeling errors, which limit their practical applications. To advance the effectiveness and robustness of model-based contact-rich in-hand manipulation, this paper proposes a novel integrated framework that mitigates these limitations. The integration involves two key aspects: 1) integrated real-time planning and tracking achieved by a hierarchical structure; and 2) joint optimization of motions and contacts achieved by integrated motion-contact modeling. Specifically, at the high level, finger motion and contact force references are jointly generated using contact-implicit model predictive control. The high-level module facilitates real-time planning and disturbance recovery. At the low level, these integrated references are concurrently tracked using a hand force-motion model and actual tactile feedback. The low-level module compensates for modeling errors and enhances the robustness of manipulation. Extensive experiments demonstrate that our approach outperforms existing model-based methods in terms of accuracy, robustness, and real-time performance. Our method successfully completes five challenging tasks in real-world environments, even under appreciable external disturbances.

Figures (21)

• Figure 1: Proposed framework for in-hand manipulation. The framework features integrated, real-time motion-contact planning and tracking. The system simultaneously plans and tracks finger motions and contact forces, enabling robust and precise manipulation in real-world scenarios. It allows the fingers to efficiently alternate between making and breaking contacts.
• Figure 2: Proposed integrated motion-contact planning and tracking framework. (A) The user inputs the desired object motion, hand grasp pose, and corresponding models. (B) The high-level real-time motion-contact planner employs contact-implicit MPC to generate motion-contact references from the initial state $\bm{x}_0$, state reference $\bm{x}_\text{ref}$, and the previous iteration's solution $\bm{X}^{*}, \bm{U}^{*}$. (C) The low-level tactile-feedback tracking controller uses tactile feedback to track these references jointly. The core algorithm is an MPC-based HFMC. (D) Together, these modules ensure robust and precise in-hand manipulation across multiple tasks.
• Figure 3: Detailed view of the proposed framework. The top figures show the high-level integrated motion-contact planning module, which generates real-time finger motions and contact information (visualizing only the index finger and forces). The top-right figure illustrates how modeling errors lead to the force-at-a-distance effect, where non-zero planned forces appear even when contact is inactive. Modeling errors can be mitigated through low-level motion-contact tracking (shown in the bottom-right figure). The bottom-left figure shows how contact tracking is achieved by deforming the command fingertip trajectory. Best viewed in color.
• Figure 4: Illustration of trajectory interpolation and shifting. DDP generates trajectories $\bm{X}^*$ at fixed intervals, shown by light green curves. The low-level module interpolates the previous solution $\bm{X}_\text{old}^{*}$ until a new one is received at $t_\text{recv}$. The interpolated trajectory is shown in dark green, and the arrow indicates discontinuity. To reduce the discontinuity, we shift the new solution to $\bm{X}_\text{new}^{*}$. The interpolated trajectory is transformed into joint commands by the low-level module. Note that due to multiple delays, the real system states $\bm{x}_\text{real}$ always lag the commanded states.
• Figure 5: Notations of the force-motion model. The finger is modeled with joint proportional-derivative (PD) control with stiffness $\bm{K}_P$ and $\bm{K}_D$. Due to controller compliance, the desired fingertip location differs from the actual location. At the contact point, the environment stiffness $\bm{K}_e$ and the finger's Cartesian stiffness $\bm{K}_r$ result in the equivalent stiffness $\bar{\bm{K}}_s$. The single contact stiffness relates the desired motions $d\bm{p}_{c,d}$ with the contact force $\bm{\lambda}_\text{ext}$. The forces and motions of all contacts are correlated due to the object displacement $\delta\bm{x}^u$. The force controlled direction $\hat{\bm{n}}$ and motion controlled directions $\{\hat{\bm{t}}_1, \hat{\bm{t}}_2\}$ are determined using the contact normal. This figure has been adapted from the conference version of this work jiang2024contact.

Theorems & Definitions (1)

• Remark