Multi-finger Manipulation via Trajectory Optimization with Differentiable Rolling and Geometric Constraints

TL;DR

The paper tackles dexterous multi-finger manipulation under fixed contact modes by castling the problem as a differentiable trajectory optimization problem. It combines sampled geometry for non-primitive finger shapes with the object's Signed Distance Field (SDF) and optimizes using Constrained Stein Variational Trajectory Optimization (CSVTO) to ensure constraint satisfaction within an MPC framework. The core contributions are a differentiable 3D finger-rolling formulation, a geometry-parametrized constraint pipeline (including contact, kinematics, wrench balance, and friction), and a benchmark with challenging tasks that reveal the benefits of accurate geometry modeling. Results in both simulation and real-world screwdriver turning and cuboid reorientation demonstrate improved object configurations and reduced dropping, highlighting robustness to sim2real gaps. This approach advances reliable, geometry-aware dexterous manipulation suitable for dynamic task scenarios without resorting to exhaustive RL training.

Abstract

Parameterizing finger rolling and finger-object contacts in a differentiable manner is important for formulating dexterous manipulation as a trajectory optimization problem. In contrast to previous methods which often assume simplified geometries of the robot and object or do not explicitly model finger rolling, we propose a method to further extend the capabilities of dexterous manipulation by accounting for non-trivial geometries of both the robot and the object. By integrating the object's Signed Distance Field (SDF) with a sampling method, our method estimates contact and rolling-related variables in a differentiable manner and includes those in a trajectory optimization framework. This formulation naturally allows for the emergence of finger-rolling behaviors, enabling the robot to locally adjust the contact points. To evaluate our method, we introduce a benchmark featuring challenging multi-finger dexterous manipulation tasks, such as screwdriver turning and in-hand reorientation. Our method outperforms baselines in terms of achieving desired object configurations and avoiding dropping the object. We also successfully apply our method to a real-world screwdriver turning task and a cuboid alignment task, demonstrating its robustness to the sim2real gap.

Figures (4)

• Figure 1: Rolling the finger can extend the flexibility of the robot and is necessary for many dexterous manipulation tasks. Parameterizing the geometry of both the robot fingers and the manipulated object is important for formulating the finger-object contacts and rolling behaviors. We propose a sampling method to approximate the geometry of the robots. Integrating it with the SDF of the object, contact-related variables such as distance between meshes, contact points, and contact normals can be estimated in a differentiable manner.
• Figure 2: (a) Pure Rolling. The blue(robot) and the green(object) rigid bodies contact at the red point $\mathbf{c}$. The contact points have velocities $\mathbf{v}_{c,r}$ and $\mathbf{v}_{c,o}$ respectively. Pure rolling happens when the contact point pair has the same velocity: $\mathbf{v}_{c,r} = \mathbf{v}_{c,o}$. (b) Projection of the kinematics constraint into the tangential space of the contact. Specifically, the constraint is satisfied if the blue and the green vector both lie on the red dashed line. (c) Variables used in the force balance. The robot wants to move from $\mathbf{q}_t$ to $\mathbf{q}_t + \hat{\mathbf{u}}_t$ but ends up at $\mathbf{q}_{t+1}$. Thus there will be an end effector force $\mathbf{f}_{i,t}$ approximately pointing in the $\mathbf{q}_t + \hat{\mathbf{u}}_t$ direction.
• Figure 3: (a) Points (shown in red) are sampled over the surface of the robot. Points closer to the object are assigned a higher weight $h_j$, which is visualized with higher saturation. The contact point $\mathbf{p}_c$ is a weighted sum of $\mathbf{p}_j$. (b) Visualization of the sampled points on an actual robot
• Figure 4: The benchmark consists of multi-finger manipulation tasks with and without extrinsic environment contacts.