Grasping a Handful: Sequential Multi-Object Dexterous Grasp Generation

TL;DR

The paper tackles sequential dexterous grasping with a hand-wide but partial set of controllable DoF. It introduces SeqGrasp to plan grasps sequentially by selecting opposition spaces and optimizing under an energy function, producing SeqDataset, a large-scale sequential grasping dataset. A diffusion-based SeqDiffuser is then trained on SeqDataset to enable fast, conditional sequential grasp generation. Across simulation and real-robot experiments, SeqGrasp and SeqDiffuser outperform the state-of-the-art MultiGrasp, with SeqDiffuser offering 750–1250x faster grasp generation, demonstrating the practicality of sequential multi-object dexterous manipulation.

Abstract

We introduce the sequential multi-object robotic grasp sampling algorithm SeqGrasp that can robustly synthesize stable grasps on diverse objects using the robotic hand's partial Degrees of Freedom (DoF). We use SeqGrasp to construct the large-scale Allegro Hand sequential grasping dataset SeqDataset and use it for training the diffusion-based sequential grasp generator SeqDiffuser. We experimentally evaluate SeqGrasp and SeqDiffuser against the state-of-the-art non-sequential multi-object grasp generation method MultiGrasp in simulation and on a real robot. The experimental results demonstrate that SeqGrasp and SeqDiffuser reach an 8.71%-43.33% higher grasp success rate than MultiGrasp. Furthermore, SeqDiffuser is approximately 1000 times faster at generating grasps than SeqGrasp and MultiGrasp. Project page: https://yulihn.github.io/SeqGrasp/.

Figures (8)

• Figure 1: Sequential multi-object grasping.
• Figure 2: Multi-object grasping visualizations for different hands. From left to right: Allegro Hand, Shadowhand, MANO.
• Figure 3: (a) Grasps using all seven oses. From left to right, first row: middle-ring, index-middle, and thumb-index, second row: ring-palm, middle-palm, index-palm, and thumb-palm. (b) Visualization of contact point candidates on Allegro Hand surface. Cyan and pink points denote palm opposition and side opposition contacts, respectively.
• Figure 4: (a) Initialization. The initial grasp configurations are randomly sampled on the expanded convex hull of the object (bottle) while the previously grasped object (ball) remains in the hand. (b) Optimization. During optimization, the grasp is incrementally refined, ultimately securing the target object using the ring-palm os.
• Figure 5: Qualitative results. For SeqGrasp (first row) and SeqDiffuser (second row), the grasp sequences are visually indicated by a color gradient, transitioning from lighter to darker shades. In contrast, for MultiGrasp (third row), the color gradient is only used to differentiate the objects. For SeqGrasp and SeqDiffuser, we only show consumed grasps, that is, when $\mathcal{OS}=\emptyset$.

Theorems & Definitions (1)

• Definition 1: Sequential multi-object grasping