Bimanual Deformable Bag Manipulation Using a Structure-of-Interest Based Neural Dynamics Model

TL;DR

A bimanual manipulation framework that leverages a graph neural network-based neural dynamics model to succinctly represent and predict the behavior of structures of interest in deformable fabric bags to enable robotic manipulators to perform precise and stable manipulation tasks focused on the SOIs.

Abstract

The manipulation of deformable objects by robotic systems presents a significant challenge due to their complex and infinite-dimensional configuration spaces. This paper introduces a novel approach to Deformable Object Manipulation (DOM) by emphasizing the identification and manipulation of Structures of Interest (SOIs) in deformable fabric bags. We propose a bimanual manipulation framework that leverages a Graph Neural Network (GNN)-based latent dynamics model to succinctly represent and predict the behavior of these SOIs. Our approach involves constructing a graph representation from partial point cloud data of the object and learning the latent dynamics model that effectively captures the essential deformations of the fabric bag within a reduced computational space. By integrating this latent dynamics model with Model Predictive Control (MPC), we empower robotic manipulators to perform precise and stable manipulation tasks focused on the SOIs. We have validated our framework through various empirical experiments demonstrating its efficacy in bimanual manipulation of fabric bags. Our contributions not only address the complexities inherent in DOM but also provide new perspectives and methodologies for enhancing robotic interactions with deformable objects by concentrating on their critical structural elements. Experimental videos can be obtained from https://sites.google.com/view/bagbot.

Figures (12)

• Figure 1: (Left) SOI Examples for different deformable object manipulation tasks, e.g., garment hanging, and robot-assistive dressing. (Right) Conceptual representation of the manifold with boundary. The manifold encompasses $\operatorname{Int} \mathcal{M}$ and $\partial \mathcal{M}$, where the local neighborhoods of points in $\operatorname{Int} \mathcal{M}$ and ${\partial \mathcal{M}}$ are homeomorphically equivalent to $\operatorname{Int} \mathbb{H}^{n}$ and $\partial \mathbb{H}^{n}$.
• Figure 2: The two robots grasp two handles of a fabric bag to manipulate the SOI (i.e., the opening rim) into the target configuration.
• Figure 3: (Left) Conceptual representation of SOI-based bimanual deformable object manipulation problem; (Right) The problem is formulated as a POMDP setting, where the SOI-related observation $\mathbf{o}^{soi}_t$ is extracted from the original observation $o_t$ and governed by $f_{\theta_\mathrm{dyn}}$.
• Figure 4: The conceptual representation of the proposed framework for bimanual deformable fabric bag manipulation using the latent SOI dynamics model.
• Figure 5: The proposed global particle sampling process: (a) Reconstructed point cloud (PCD) $\mathcal{P}{\text{full}}$; (b) SOI point cloud after HSV filtering $\mathcal{P}{\text{soi}}$; (c) SOI point cloud after down-sampling and outlier removal; (d) Normal generation; (e) Reconstructed SOI surface using ball pivoting; (f) SOI particles after uniform sampling.