AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulation

TL;DR

AdaptiGraph tackles the challenge of modeling and controlling deformable objects with unknown properties by learning a material-conditioned GBND and online few-shot adaptation. It conditions on discrete material type $M$ and continuous property $\phi$, and uses inverse optimization to estimate properties from interactions, enabling accurate forward dynamics and planning via MPC/MPPI. The approach demonstrates improved prediction accuracy and manipulation success across ropes, granular media, cloth, and rigid boxes in both simulation and real-world experiments. This work advances generalization in deformable object manipulation by integrating explicit material conditioning with learnable dynamics and online system identification.

Abstract

Predictive models are a crucial component of many robotic systems. Yet, constructing accurate predictive models for a variety of deformable objects, especially those with unknown physical properties, remains a significant challenge. This paper introduces AdaptiGraph, a learning-based dynamics modeling approach that enables robots to predict, adapt to, and control a wide array of challenging deformable materials with unknown physical properties. AdaptiGraph leverages the highly flexible graph-based neural dynamics (GBND) framework, which represents material bits as particles and employs a graph neural network (GNN) to predict particle motion. Its key innovation is a unified physical property-conditioned GBND model capable of predicting the motions of diverse materials with varying physical properties without retraining. Upon encountering new materials during online deployment, AdaptiGraph utilizes a physical property optimization process for a few-shot adaptation of the model, enhancing its fit to the observed interaction data. The adapted models can precisely simulate the dynamics and predict the motion of various deformable materials, such as ropes, granular media, rigid boxes, and cloth, while adapting to different physical properties, including stiffness, granular size, and center of pressure. On prediction and manipulation tasks involving a diverse set of real-world deformable objects, our method exhibits superior prediction accuracy and task proficiency over non-material-conditioned and non-adaptive models. The project page is available at https://robopil.github.io/adaptigraph/ .

Figures (12)

• Figure 1: Motivation. Objects made from different materials can exhibit distinct behaviors under interaction. Even within the same object category, varying physical parameters like stiffness can lead to different behaviors. Examples shown here include handling cotton rope and cable, as well as arranging granular piles such as coffee beans and toy blocks. Although the initial configuration and action are the same, different physical parameters result in distinct final states, necessitating the need for online adaptation for effective manipulation. To this end, we introduce AdaptiGraph, a unified graph-based neural dynamics framework for real-time modeling and control of various materials with unknown physical properties. AdaptiGraph integrates a physical property-conditioned dynamics model with online physical property estimation. Our framework enables robots to adaptively manipulate diverse objects with varying physical properties and dynamics.
• Figure 2: Overview of proposed framework: AdaptiGraph.(a) Our graph-based dynamics model $f$ is conditioned on the discrete material type and continuous physical parameters $\phi$. $\phi$ is encoded as the node features, which will be propagated and updated in the model training process. Our model can accurately predict the future state $\hat{z}_{t+1}$ for a variety of objects with different physical properties. (b) Our framework performs physical property estimation for few-shot adaptation. This is achieved through an inverse optimization process to estimate the optimal physical parameters as predicted by the learned dynamics model $f$. The optimal physical parameter $\phi^*$ is identified by minimizing the cost function, which is defined as the Chamfer Distance between the predicted graph state and the actual future graph state.
• Figure 3: Real-world setup.(a) Our study involves 22 objects categorized into four types of materials, each with distinct physical characteristics: (i) 9 varieties of ropes, such as cotton ropes and cables, (ii) 9 granular materials, including items like toy blocks and coffee beans, (iii) 5 pieces of cloth made from different fabrics like cotton and synthetic fibers, (iv) 2 boxes of varying shapes, whose centers of pressure we alter by placing weights inside them. (b) The dashed white circles show four calibrated RGB-D cameras mounted at four corners of the table. The robot is outfitted with specialized end effectors to interact with the objects in its operational area. (c) We employ three different tools for specific tasks: (1) a flat pusher for granular piles gathering, (2) a cylindrical pusher for pushing rigid boxes and straightening ropes, (3) an xArm gripper for cloth relocating.
• Figure 4: Qualitative results on dynamics prediction. We conduct qualitative comparisons to assess the performance of our method against the baseline of a GNN without adaptation, focusing on the one-step prediction of dynamics across eight objects within four distinct material categories exhibiting varying extreme physical properties. The results, delineated by red dashed boxes, demonstrate that our approach surpasses the baseline in accurately capturing the variations in dynamics that arise due to differences in the objects' physical properties.
• Figure 5: Quantitative results on dynamics prediction. We validate our model's effectiveness on a test set of 200 objects with distinct physical properties for each material type in simulation. Across all types of materials, our approach surpasses the baseline with respect to both the precision and consistency of predictions.