Trajectory Optimization for In-Hand Manipulation with Tactile Force Control

TL;DR

This work tackles robust in-hand rolling of small objects with a Shadow Hand equipped with Magnetic Tactile Sensors. It introduces an NLP-based offline trajectory planner for multi-finger contact dynamics, augmented by a tactile-based state estimator and a cascade finger force/position controller that accounts for tendon compliance. Experimental results show that adding force control substantially improves grip stability and rolling accuracy, achieving 7/10 successful rolls compared to open-loop failures. The framework enables precise, robust manipulation in tight spaces using compact tactile sensing and model-based planning.

Abstract

The strength of the human hand lies in its ability to manipulate small objects precisely and robustly. In contrast, simple robotic grippers have low dexterity and fail to handle small objects effectively. This is why many automation tasks remain unsolved by robots. This paper presents an optimization-based framework for in-hand manipulation with a robotic hand equipped with compact Magnetic Tactile Sensors (MTSs). The small form factor of the robotic hand from Shadow Robot introduces challenges in estimating the state of the object while satisfying contact constraints. To address this, we formulate a trajectory optimization problem using Nonlinear Programming (NLP) for finger movements while ensuring contact points to change along the geometry of the fingers. Using the optimized trajectory from the solver, we implement and test an open-loop controller for rolling motion. To further enhance robustness and accuracy, we introduce a force controller for the fingers and a state estimator for the object utilizing MTSs. The proposed framework is validated through comparative experiments, showing that incorporating the force control with compliance consideration improves the accuracy and robustness of the rolling motion. Rolling an object with the force controller is 30\% more likely to succeed than running an open-loop controller. The demonstration video is available at https://youtu.be/6J_muL_AyE8.

Figures (8)

• Figure 1: Rolling an object using a robotic hand. The First finger, with 4 degrees of freedom (DoF), and the thumb, with 5 DoF, are equipped with Magnetic Tactile Sensors (MTSs).
• Figure 2: The architecture of the proposed framework. The model-based trajectory optimization generates offline finger reference motions for rolling, and the finger force controller tracks these motions using tactile-based state estimation for rolling. This ensures robust and stable grasping of a given object. Furthermore, the compliant behavior of the hand compromises the performance of the finger controller; thus, our framework also includes compliance compensation, which enhances the performance of the reference tracking control.
• Figure 3: Illustration of the rolling motion of a cylinder manipulated by two fingers. The upper finger represents the First finger and the bottom finger represents the Thumb finger.
• Figure 4: Illustration of the contact point estimator with continuous contact input.
• Figure 5: Ground truth ($\varphi$) and estimated orientation ($\hat{\varphi}$) of the object during rolling motion. The row (a) of the figure shows the object's orientation under an open-loop controller, while the row (b) shows the orientation with an additional force controller.