Learning Contact Dynamics through Touching: Action-conditional Graph Neural Networks for Robotic Peg Insertion

TL;DR

A learnable physics-based predictive model that provides accurate motion and force-torque prediction of the robot end effector in contact-rich manipulation is presented, enabling action-conditional predictions for control and state estimation in the context of robotic peg insertion.

Abstract

We present a learnable physics-based predictive model that provides accurate motion and force-torque prediction of the robot end effector in contact-rich manipulation. The proposed model extends the state-of-the-art GNN-based simulator (FIGNet) with novel node and edge types, enabling action-conditional predictions for control and state estimation in the context of robotic peg insertion. Our model learns in a self-supervised manner, using only joint encoder and force-torque data while the robot is touching the environment. In simulation, the MPC agent using our model matches the performance of the same controller with the ground truth dynamics model in a challenging peg-in-hole task, while in the real-world experiment, our model achieves a 50$\%$ improvement in motion prediction accuracy and 3$\times$ increase in force-torque prediction precision over the baseline physics simulator. Finally, we apply the model to track the robot end effector with a particle filter during real-world peg insertion, demonstrating a practical application of its predictive accuracy.

Figures (17)

• Figure 1: Simulation pipeline overview. Cf. text for details about (1) Graph construction, (2)EPD stack, and (3) Post-processing.
• Figure 2: Action-conditional edges between the world nodes and the tool mesh nodes (top). Face interaction mesh-mesh edges are created when two faces ($\mathcal{F}_s$ and $\mathcal{F}_r$) fall within a predefined collision sphere (bottom).
• Figure 3: The training scene consists of a UR10e robot with three random tool shapes and three to six box obstacles (left). Peg insertion task scene with different tools and their matching slot geometries (right).
• Figure 4: Results on the peg insertion tasks with different geometries. Performance comparisons of the MPC agent with the learned and baseline models are shown. The fine-tuned model is labeled as Act-FIGNet-F, while the model trained on the augmented dataset is labeled as Act-FIGNet-A.
• Figure 5: (a) Bota SenseOne F/T sensor; (b) 3D printed tools; (c) Hexagonal slot with small clearance; (d) Setup for the real-world experiment with UR10e; (e) The system contains only the tool and world body represented with triangle meshes; (f) In MuJoCo, the tool is modeled as a floating free body with 3 hinge joints and a ball joint.