Estimating Deformable-Rigid Contact Interactions for a Deformable Tool via Learning and Model-Based Optimization

TL;DR

This work tackles dexterous manipulation with deformable tools by combining a learned latent-dynamics model that predicts rigid-object motion and tool-induced contacts with a physics-informed Contact Quadratic Program (CQP) that recovers environment forces under quasi-static equilibrium and Coulomb friction. The method jointly infers intrinsic motions and both deformable-tool and environment contacts, using multimodal encoders and a latent dynamics backbone to forecast future states, followed by a CQP to enforce physical constraints. Trained in simulation with diverse geometries and material properties, the approach demonstrates accurate tool-contact and object-motion predictions and favorable force estimates, with successful sim-to-real transfer on a Franka robot and force-tracking capabilities. The results suggest a viable path toward reliable, deformable-influenced manipulation and planning, leveraging both data-driven predictions and principled physics for contact reasoning in dexterous tasks.

Abstract

Dexterous manipulation requires careful reasoning over extrinsic contacts. The prevalence of deforming tools in human environments, the use of deformable sensors, and the increasing number of soft robots yields a need for approaches that enable dexterous manipulation through contact reasoning where not all contacts are well characterized by classical rigid body contact models. Here, we consider the case of a deforming tool dexterously manipulating a rigid object. We propose a hybrid learning and first-principles approach to the modeling of simultaneous motion and force transfer of tools and objects. The learned module is responsible for jointly estimating the rigid object's motion and the deformable tool's imparted contact forces. We then propose a Contact Quadratic Program to recover forces between the environment and object subject to quasi-static equilibrium and Coulomb friction. The results is a system capable of modeling both intrinsic and extrinsic motions, contacts, and forces during dexterous deformable manipulation. We train our method in simulation and show that our method outperforms baselines under varying block geometries and physical properties, during pushing and pivoting manipulations, and demonstrate transfer to real world interactions. Video results can be found at https://deform-rigid-contact.github.io/.

Figures (8)

• Figure 1: We present a method that estimates motions and forces during dexterous manipulation with a deformable tool. Our proposed method takes in object information (geometry, center of mass, mass, and friction) and sensing from the deforming tool (partial point cloud). It estimates the block motion, tool force the deforming tool enacts on the block, and the rigid contact forces between block and (known) environment, given the robot actions. On the bottom left, we show one-step predictions from our method for a real pivoting execution.
• Figure 2: Overview of our proposed method. Our method takes in partial views of the tool via point clouds along with the current object state and parameters and encodes to a learned latent space. Our latent dynamics module then rolls out in the latent space given actions. Finally, our model regresses a) object motion and b) tool summary contact point and force. We use the object motion to determine environment contacts and solve a Contact Quadratic Program (CQP) of our design which resolves the environment contact forces subject to friction and quasi-static equilibrium.
• Figure 3: Qualitative Results. Left: We compare our methods one-step predictions (solid) to the ground truth (semi-transparent) object motion, tool force, and environment forces across several manipulation trajectories. The friction cones for each environment contact is shown in gray at the environment point of contact. Our method shows high-fidelity predictions for a) varying object geometries and physical parameters, and b) different interaction primitives (pivoting vs. pushing). Right: We show one-step predictions for real robot executions. The predictions match the observed motion and show qualitatively realistic forces (see Sec. \ref{['sec:real_results']} for quantitative results).
• Figure 4: Tool Contact Force (purple) and Contact Point Location (green) error for our proposed model, plotted by prediction horizon time step. Error bars indicate one half standard deviation.
• Figure 5: Baseline comparison of our method on test simulated interactions. For (a) and (b) we compare to a Rigid baseline that predicts block motion as if rigidly attached to the tool. For (c) we compare our model-based optimization for extrinsic contact recovery to directly predicting it from an MLP. Error bars indicate one half standard deviation.