Robotic in-hand manipulation with relaxed optimization

TL;DR

This paper addresses dexterous in-hand manipulation by learning a dictionary of motion primitives from human demonstrations and using a relaxed optimization to compose fingertip and object trajectories. The approach encodes trajectories as v = [P(1) … P(N)] and uses non-negative matrix factorization to extract primitives, enabling reconstruction via v = W h. By optimizing start/end pose accuracy with a learned primitive dictionary, the method implicitly respects stability and contact constraints, achieving fast trajectory generation (around a few hundred milliseconds) on a Shadow Hand platform while maintaining at least two fingertip contacts. Experiments with cube and cylinder objects show sub-millimeter fingertip errors and small orientation deviations, with significantly lower training and planning times than analytical or data-driven baselines. The work suggests broader applicability to non-anthropomorphic hands by focusing on fingertip poses rather than detailed kinematics, paving the way for practical, human-inspired in-hand manipulation in robotics.

Abstract

Dexterous in-hand manipulation is a unique and valuable human skill requiring sophisticated sensorimotor interaction with the environment while respecting stability constraints. Satisfying these constraints with generated motions is essential for a robotic platform to achieve reliable in-hand manipulation skills. Explicitly modelling these constraints can be challenging, but they can be implicitly modelled and learned through experience or human demonstrations. We propose a learning and control approach based on dictionaries of motion primitives generated from human demonstrations. To achieve this, we defined an optimization process that combines motion primitives to generate robot fingertip trajectories for moving an object from an initial to a desired final pose. Based on our experiments, our approach allows a robotic hand to handle objects like humans, adhering to stability constraints without requiring explicit formalization. In other words, the proposed motion primitive dictionaries learn and implicitly embed the constraints crucial to the in-hand manipulation task.

Figures (6)

• Figure 1: The Shadow Hand manipulating a cylinder. The reference frame for the palm ($R_{P}$) is shown, with the $x$, $y$, and $z$ axes in orange, green, and blue arrows, respectively.
• Figure 2: Six Flex-3 cameras arranged on a one-meter cube frame.
• Figure 3: On the left, the markers are placed on two different objects, while on the right, the markers are placed on a hand. The back of the hand and the two objects have multiple markers on rigid supports for tracking their orientation.
• Figure 4: The picture shows the platform used for the experiments. This includes the Shadow Hand, a mechanical arm, and 5 force sensors placed on the fingertips (AT Nano17).
• Figure 5: This series of images shows the manipulation of a cube over time. First, the Shadow Hand rotates the object around the y-axis. Then, the Shadow Hand translates the object along the y-axis. Finally, the Shadow Hand performs a composite motion to return the object to its initial pose. (The system of reference $R_{P}$ is the same considered in Fig. \ref{['fig:example']}).