DeRi-Bot: Learning to Collaboratively Manipulate Rigid Objects via Deformable Objects

TL;DR

This work tackles the challenge of manipulating rigid objects through deformable, rope-like links in heterogeneous soft-rigid systems. It introduces DeRi-Bot, a data-driven framework comprising an Action Prediction Network (APN) and a Configuration Prediction Network (CPN) that together handle action selection and outcome forecasting under rope stochasticity. The method supports arbitrary numbers of agents or human partners via asynchronous decoupling and a sampling-augmented, forecast-guided workflow, with demonstrated robustness across multi-agent and human-robot collaboration scenarios in both simulation and real-world settings. Key contributions include the first end-to-end learning framework for soft-rigid, rope-mediated manipulation, a Gaussian action augmentation around APN outputs, and a visual foresight module (CPN) for action selection, enabling scalable and transferable manipulation of rigid objects through deformable connectors.

Abstract

Recent research efforts have yielded significant advancements in manipulating objects under homogeneous settings where the robot is required to either manipulate rigid or deformable (soft) objects. However, the manipulation under heterogeneous setups that involve both rigid and one-dimensional (1D) deformable objects remains an unexplored area of research. Such setups are common in various scenarios that involve the transportation of heavy objects via ropes, e.g., on factory floors, at disaster sites, and in forestry. To address this challenge, we introduce DeRi-Bot, the first framework that enables the collaborative manipulation of rigid objects with deformable objects. Our framework comprises an Action Prediction Network (APN) and a Configuration Prediction Network (CPN) to model the complex pattern and stochasticity of soft-rigid body systems. We demonstrate the effectiveness of DeRi-Bot in moving rigid objects to a target position with ropes connected to robotic arms. Furthermore, DeRi-Bot is a distributive method that can accommodate an arbitrary number of robots or human partners without reconfiguration or retraining. We evaluate our framework in both simulated and real-world environments and show that it achieves promising results with strong generalization across different types of objects and multi-agent settings, including human-robot collaboration.

Figures (7)

• Figure 1: A real-world scenario of moving a rigid object (tree trunk) via soft objects (ropes).
• Figure 2: A demonstration of DeRi-Bot collaborating with a human in the real world in moving the rigid brown box to its target position given by a green marker.
• Figure 3: An iteration of DeRi-Bot workflow. The APN model takes as input the state observation to predict an action. The Sampler generates more actions around the output of APN. The CPN model predicts the corresponding next states of the rigid object of all the actions. The framework picks the best action that yields the minimum distance between the object and the target position. This figure shows such a process for the robot on the top. However, by design, all the robots' proposed actions will be compared together for the selection of the robot and its best action. Furthermore, note that we add the target in yellow to the visualization of CPN outputs to illustrate the effect of different actions.
• Figure 4: An example of a dual-bot setup in moving the green box to its target position indicated by the purple marker.
• Figure 5: An example of the quad-bot collaboration process in moving the green box to its purple target. All robots propose their actions, and the robot yielding the minimum distance to the target is selected based on the CPN visual foresight.