Robotic In-Hand Manipulation for Large-Range Precise Object Movement: The RGMC Champion Solution

TL;DR

The work tackles precise, large-range in-grasp object movement under a constant stable grasp, proposing a geometry-free trajectory optimization framework that avoids object geometries and pretraining. By allowing fingertip rolling, planning full trajectories, and applying a closed-loop re-planning scheme, the method expands the reachable in-hand space and improves robustness. It achieves the RGMC 2024 in-hand manipulation championship, demonstrates around 5 mm average error over 40 waypoints in a $5 \times 5 \times 5$ cm cube, and generalizes well to several novel everyday objects, including curved and lightweight items. The approach is practical, implementable on common hardware with full-actuated hands, and offers a principled balance between accuracy, reach, and robustness for real-world dexterous manipulation.

Abstract

In-hand manipulation using multiple dexterous fingers is a critical robotic skill that can reduce the reliance on large arm motions, thereby saving space and energy. This letter focuses on in-grasp object movement, which refers to manipulating an object to a desired pose through only finger motions within a stable grasp. The key challenge lies in simultaneously achieving high precision and large-range movements while maintaining a constant stable grasp. To address this problem, we propose a simple and practical approach based on kinematic trajectory optimization with no need for pretraining or object geometries, which can be easily applied to novel objects in real-world scenarios. Adopting this approach, we won the championship for the in-hand manipulation track at the 9th Robotic Grasping and Manipulation Competition (RGMC) held at ICRA 2024. Implementation details, discussion, and further quantitative experimental results are presented in this letter, which aims to comprehensively evaluate our approach and share our key takeaways from the competition. Supplementary materials including video and code are available at https://rgmc-xl-team.github.io/ingrasp_manipulation .

Figures (14)

• Figure 1: In-grasp object movement task, where the goal is to manipulate the in-hand object to a desired pose (position) using only finger motions within a stable grasp. (a) Scene of the Robotic Grasping and Manipulation Competition (RGMC) at ICRA 2024, where we won the championship for this task. (b)(c) Precisely moving the object to the desired position in a large in-hand space.
• Figure 2: Our hardware setup for the competition, comprising a Leap Hand with soft silicone fingertips, a top-view RGB camera, and an in-hand object with an AprilTag marker. Each silicone fingertip comprises an inner 3-D printed base and an outer silicone layer.
• Figure 3: Formulation of the in-grasp object movement. The objective is to find a hand joint trajectory to move the object, starting from $\bm T_{\text{o}, 0}$ and reaching $\bm T_{\text{o}, \text{d}}$ at time step $T$ while maintaining a constant stable grasp.
• Figure 4: Our pipeline for the RGMC. After the initial grasping, our solution iteratively plans a path from the current state to the goal through trajectory optimization and executes it until certain conditions are satisfied. Then, it switches to reach the next waypoint, during which the hand first returns to the initial state and then moves the object to the next goal.
• Figure 5: Experiments of in-grasp object movement with various objects, in which the objects continuously reach the eight corners of a $5\times5\times5$ (cm) cubic space. (a) The known cylinder object provided by the competition organizer. (b) The novel everyday objects used in our experiments, including a thick cylinder lid, box, presenter remote, and screwdriver. For each object, the images in the first row are from the top-view camera used for object pose tracking, where the red points and green circles represent the AprilTag centers and the desired positions, respectively; the images in the second row are from another camera used only for visualization. More manipulation processes are shown in the https://rgmc-xl-team.github.io/ingrasp_manipulation.