Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation

Abstract

We address the challenge of learning to manipulate deformable objects with unknown dynamics. In non-rigid objects, the dynamics parameters define how they react to interactions -- how they stretch, bend, compress, and move -- and they are critical to determining the optimal actions to perform a manipulation task successfully. In other robotic domains, such as legged locomotion and in-hand rigid object manipulation, state-of-the-art approaches can handle unknown dynamics using Rapid Motor Adaptation (RMA). Through a supervised procedure in simulation that encodes each rigid object's dynamics, such as mass and position, these approaches learn a policy that conditions actions on a vector of latent dynamic parameters inferred from sequences of state-actions. However, in deformable object manipulation, the object's dynamics not only includes its mass and position, but also how the shape of the object changes. Our key insight is that the recent ground-truth particle positions of a deformable object in simulation capture changes in the object's shape, making it possible to extend RMA to deformable object manipulation. This key insight allows us to develop RAPiD, a two-phase method that learns to perform real-robot deformable object mobile manipulation by: 1) learning a visuomotor policy conditioned on the object's dynamics embedding, which is encoded from the object's privileged information in simulation, such as its mass and ground-truth particle positions, and 2) learning to infer this embedding using non-privileged information instead, such as robot visual observations and actions, so that the learned policy can transfer to the real world. On a mobile manipulator with 22 degrees of freedom, RAPiD enables over 80%+ success rates across two vision-based deformable object mobile manipulation tasks in the real world, under various object dynamics, categories, and instances.

Figures (6)

• Figure 1: RAPiD: $\operatorname{\text{\underline{R}apid \underline{A}daptation of \underline{P}art\underline{i}cle \underline{D}ynamics}}$ (Left) is a method for learning to perform real-world deformable object mobile manipulation tasks by inferring and adapting to the unknown dynamics of deformable objects in real-time. Top Right: In simulation, RAPiD first learns a visuomotor policy to achieve a deformable object mobile manipulation task, such as bowl-covering, by inferring the deformable object's dynamics from the robot's visual observations and actions. Bottom Right: RAPiD then deploys the learned visuomotor policy directly to the real world to achieve the deformable object mobile manipulation task, under unseen object dynamics, instances, categories, and lighting conditions, using only onboard sensor signals.
• Figure 2: The RAPiD Method. Below we explain training (top and middle) in simulation and deploying in the real world (bottom) a deformable object mobile manipulation solution with RAPiD. Training with RAPiD (top and middle) is a two-step procedure. In the first step (top), the agent is asked to solve a simulated version of the task by learning a visuomotor policy and two encoders: a Shape Encoder, $\mu_{s}$, and a Dynamics Encoder, $\mu_{d}$, using privileged state information of the object from the simulator, such as mass and particle positions. In the second step (middle), RAPiD frees itself from privileged simulator information by learning a new Shape Adaptation module, $\phi_{s}$, and Dynamics Adaptation module $\phi_{d}$, using L1-losses to replace the Shape Encoder and the Dynamics Encoder, respectively. At test time (bottom), both the Shape and Dynamics adaptation modules and the visuomotor policy are directly deployed to the real world, with the adaptation modules updating the Shape Embedding $\hat{z}_t^{s}$ and Dynamics Embeddings $\hat{z}_t^{d}$ to the visuomotor policy once every 5 timesteps.
• Figure 3: Simulated and Real-World Version of the Two Tasks: 1D_Inserting (Left Half) and 2D_Covering (Right Half). Each task is trained in simulation (left image of each half) and then deployed directly to the real world (right image of each half) under unseen objects, environments, and lighting conditions. By interacting with deformable objects across a broad range of instances, geometries, and categories in simulation, RAPiD successfully learns a visuomotor policy that adapts to and excels at manipulating real-world deformable objects of unseen dynamics, instances, and categories on a real bimanual mobile manipulator.
• Figure 4: All unseen real-world deformable objects used for the 1D_Inserting (left) and 2D_Covering (right) mobile manipulation tasks, respectively. 1D categories (left): belt, hammer, marker, banana, HDMI adapter, GPU connector, jumping rope, ethernet cable, zip tie, rake, mouse, screwdriver, stick, rubber, velvet rope, VGA adapter, anchor rope, garden rope, ribbon, fan. 2D categories (right): polyester bag, pants, towel, plastic bag, face mask, paper bag, glove, shorts, cap, lid, sock, soft cuff, cardboard, envelope, pouch, wallet, eye-mask, t-shirt, sponge, wool hat. Learning from deformable objects across a wide range of instances in simulation enables RAPiD to zero-shot transfer to real-world objects under unseen dynamics, instances, and categories.
• Figure 5: RAPiD's Dynamics Adaptive Behavior in1D_Inserting (Top) and 2D_Covering (Bottom) tasks. Top Left:RAPiD hangs the rope over the top of the container before lowering it. Top Right:RAPiD flips the gripper upside down before performing a direct adapter insertion. Bottom Left:RAPiD moves the robot arm sideways and sweeps a towel horizontally from right to center to cover the bowl. Bottom Right: RAPiD moves the arm vertically and places a hat directly on top of the bowl. This shows RAPiD's ability to infer different object dynamics and adapt actions accordingly to achieve the deformable object mobile manipulation tasks.