Learning Particle Dynamics Subject to Rigid Body Manipulations Using Graph Neural Networks

TL;DR

The paper presents a multi-graph GNN framework that learns liquid dynamics under dynamically moving rigid bodies, addressing limitations of prior static or simple-geometry simulators. It introduces BVH-based collision handling and separate node sets for liquids, objects, and surface meshes, enabling accurate, generalizable simulations and gradient-based control. The approach generalizes to unseen geometries and supports tasks like stirring, scooping, and multi-source pouring, with competitive or superior performance against baselines. Demonstrating differentiable simulation, the work enables MPC and highlights practical potential for robotics and design tasks requiring underactuated fluid–rigid interactions.

Abstract

Simulating particle dynamics with high fidelity is crucial for solving real-world interaction and control tasks involving liquids in design, graphics, and robotics. Recently, data-driven approaches, particularly those based on graph neural networks (GNNs), have shown progress in tackling such problems. However, these approaches are often limited to learning fluid behavior in static free-fall environments or simple manipulation settings involving primitive objects, often overlooking complex interactions with dynamically moving kinematic rigid bodies. Here, we propose a GNN-based framework designed from the ground up to learn the dynamics of liquids under rigid body interactions and active manipulations, where particles are represented as graph nodes and particle-object collisions are handled using surface representations with the bounding volume hierarchy (BVH) algorithm. Our approach accurately captures fluid behavior in dynamic settings and can also function as a simulator in static free-fall environments. Despite being trained on single-object manipulation tasks, our model generalizes effectively to environments with novel objects and novel manipulation tasks. Finally, we show that the learned dynamics can be leveraged to solve control and manipulation tasks using gradient-based optimization methods.

Figures (12)

• Figure 1: Rollouts from our proposed model on novel scenarios. (a) Successful simulation of pouring from the complex Utah teapot. (b) Simultaneous manipulation of two jugs, demonstrating multi-object control. The blue liquid streams, originating from each jug, collide and merge realistically. The red and green particles represent the predicted trajectories for each jug if simulated independently.
• Figure 2: Overview of our network architecture and graph encoding scheme. The figure shows the graph connectivity of a liquid particle with its neighboring particles and a Cup object. Note that, for illustration purposes only, the closest point on the surface of the cup to the particle is shown as $\mathcal{V}^{O}$. In reality, the object node $\mathcal{V}^{O}$ is virtual and located at the object's center of mass.
• Figure 3: Example rollouts of our Full-model on the held-out test set.
• Figure 4: Comparison with the baseline models. GNS and MGN struggle to handle particle dynamics with novel geometry, while our method generalises without any additional training.
• Figure 5: Rollouts of our Full-model showing successful Scooping (left) and Stirring (right) manipulation tasks