A Learning-based Controller for Multi-Contact Grasps on Unknown Objects with a Dexterous Hand

TL;DR

This work tackles robust multi-contact grasping on unknown objects by introducing a learning-based, torque-driven controller that estimates the external wrench from joint torques and predicts torque commands to counteract it using an elastic contact model and a joint-level impedance controller. Two neural networks, one for wrench estimation and one for torque prediction, enable real-time operation without requiring explicit contact information or exact contact geometry; a novel minimal-wrench loss mitigates wrench-hallucination in ambiguous scenarios. In simulation, the approach stabilizes 83.1% of grasps under up to 10 N wrenches and outperforms two baselines in efficiency and object stability, while real-robot experiments on the DLR-Hand II achieve a 6 ms cycle time at 40 Hz. The results demonstrate practical applicability for autonomous dexterous manipulation of unknown objects, with potential for broader impact in robotic grasping and manipulation tasks.

Abstract

Existing grasp controllers usually either only support finger-tip grasps or need explicit configuration of the inner forces. We propose a novel grasp controller that supports arbitrary grasp types, including power grasps with multi-contacts, while operating self-contained on before unseen objects. No detailed contact information is needed, but only a rough 3D model, e.g., reconstructed from a single depth image. First, the external wrench being applied to the object is estimated by using the measured torques at the joints. Then, the torques necessary to counteract the estimated wrench while keeping the object at its initial pose are predicted. The torques are commanded via desired joint angles to an underlying joint-level impedance controller. To reach real-time performance, we propose a learning-based approach that is based on a wrench estimator- and a torque predictor neural network. Both networks are trained in a supervised fashion using data generated via the analytical formulation of the controller. In an extensive simulation-based evaluation, we show that our controller is able to keep 83.1% of the tested grasps stable when applying external wrenches with up to 10N. At the same time, we outperform the two tested baselines by being more efficient and inducing less involuntary object movement. Finally, we show that the controller also works on the real DLR-Hand II, reaching a cycle time of 6ms.

Figures (10)

• Figure 1: The controller applied to a grasp on the YCB bleach bottle. The applied external wrench (red arrow) gets estimated and then the commended torques are adjusted. Here, two joints are visualized with their applied torque before (green) and after (orange) the external wrench is applied.
• Figure 2: Block diagram of our proposed controller. The controller is built on a joint-level impedance controller which commands the underlying torque controller based on the given desired joint angles. The core of our controller consists of the estimation of the external wrench based on the measured joint torques and the subsequent prediction of the desired joint angles which are necessary to counter the external wrench.
• Figure 3: The DLR-Hand II Butterfass2001 which we use in our experiments. Each red dot represents a potential contact point. The finger on the right shows the four joints ($q_3$ and $q_4$ are coupled: $q_3 = q_4$)
• Figure 4: Architectures of both used networks. The object, represented as a voxelgrid, is encoded using 3D convolutional layers and afterward concatenated with the additional one-dimensional input. A four-layer MLP estimates the external wrench or respectively the desired torques in the end.
• Figure 5: Qualitative examples of the wrench estimation. Shows the ground truth external wrench in green and the wrench estimated by the network in red.