Learning to Move Objects with Fluid Streams in a Differentiable Simulation

TL;DR

The paper tackles contactless manipulation of objects using fluid streams by leveraging differentiable physics to train a neural controller that observes only the object's state. It employs an 8×8 grid of vertical emitters and trains a recurrent convolutional network to generate emitter velocities, achieving robust, generalizable control for hovering, targeted displacement, and multi-object separation. A key contribution is a sample-efficient DP-based training framework that also emphasizes energy efficiency through a loss term, enabling controllers that work well beyond training horizons. The work suggests practical potential for real-world fluid-based manipulation and motivates future integration with vision systems to observe object state in real devices.

Abstract

We introduce a method for manipulating objects in three-dimensional space using controlled fluid streams. To achieve this, we train a neural network controller in a differentiable simulation and evaluate it in a simulated environment consisting of an 8x8 grid of vertical emitters. By carrying out various horizontal displacement tasks such as moving objects to specific positions while reacting to external perturbations, we demonstrate that a controller, trained with a limited number of iterations, can generalise to longer episodes and learn the complex dynamics of fluid-solid interactions. Importantly, our approach requires only the observation of the manipulated object's state, paving the way for the development of physical systems that enable contactless manipulation of objects using air streams.

Figures (7)

• Figure 1: Controlled displacement of simulated objects. The solid lines in the top left shows the trajectory of the objects as they are moved by the simulated fluid. The fluid is emitted from an $8\times8$ grid, and is visualised as a grey plume. Animations can be found in the accompanied https://www.youtube.com/watch?v=sft0MH_pk9w.
• Figure 2: Differentiable control diagram. Shown is one time step of the controlled simulation. The differentiable simulator (bottom) operates on the velocity field $\mathbf{v}$ containing fluid velocity vectors for each spatial location at a time step $k$, the object's 3D position $\mathbf{x}$, and the control signal $\mathbf{u}$, corresponding to the emitter velocity. The neural network controller (top) receives the object's position from the simulator at the previous time step ($k-1$), its own hidden state $\mathbf{h}$, and the goal position of the controlled objects in order to update its hidden state and the control signal.
• Figure 3: Neural network architecture. We use a recurrent convolutional neural network that operates on an input grid of $8\times8$ fluid emitters with a corresponding feature map of $M_{\theta}$ elements, and a hidden state of size $M_h$. Dropout is applied to the hidden state to enhance generalisation to longer simulations. Two output heads performing linear projection are attached to the last layer, producing emitter velocities (range-limited via sigmoid activation) and an updated hidden state for the next time step by applying a scaled residual connection.
• Figure 4: Object position and velocity representation. (a) Object soft mask rendered onto an $8\times8$ grid. A brighter shade indicates greater coverage of the object within a grid cell. (b) Velocity representation derived by subtracting two object masks displaced by velocity. Bright colours indicate positive values, dark colours represent negative values, and zero is depicted in grey.
• Figure 5: Task: maintain the position of the object without side wind (top) and with side wind (bottom). Left side: trajectories of several objects in the $xy$ plane. The target position is marked with a "+" sign. Right side: deviation from the target throughout the simulation time. The deviation is evaluated as the Euclidean distance to the desired location. Trajectory colours between left and right plots are set to match.