Make a Donut: Hierarchical EMD-Space Planning for Zero-Shot Deformable Manipulation with Tools

TL;DR

The paper tackles deformable object manipulation without demonstrations by introducing a demonstration-free hierarchical planning framework that leverages large language models to decompose complex tasks into stages and generate Python-based subgoal point clouds. A single-tool, closed-loop planner using Differentiable Physics in Earth Mover's Distance space (DiffPhysics-P2P) then iteratively moves toward each subgoal, enabling zero-shot execution of long-horizon tasks. Key contributions include a novel LLM-guided multi-stage plan with tool-specific subgoals and a DiffPhysics-P2P-based MPC that provides one-to-one point-cloud correspondences for effective gradient-based control, validated in dough manipulation benchmarks and real-robot experiments. The approach demonstrates strong generalization to unseen tasks, outperforms baselines without task-specific training, and shows practical sim-to-real transfer, marking a significant advance in zero-shot deformable manipulation.

Abstract

Deformable object manipulation stands as one of the most captivating yet formidable challenges in robotics. While previous techniques have predominantly relied on learning latent dynamics through demonstrations, typically represented as either particles or images, there exists a pertinent limitation: acquiring suitable demonstrations, especially for long-horizon tasks, can be elusive. Moreover, basing learning entirely on demonstrations can hamper the model's ability to generalize beyond the demonstrated tasks. In this work, we introduce a demonstration-free hierarchical planning approach capable of tackling intricate long-horizon tasks without necessitating any training. We employ large language models (LLMs) to articulate a high-level, stage-by-stage plan corresponding to a specified task. For every individual stage, the LLM provides both the tool's name and the Python code to craft intermediate subgoal point clouds. With the tool and subgoal for a particular stage at our disposal, we present a granular closed-loop model predictive control strategy. This leverages Differentiable Physics with Point-to-Point correspondence (DiffPhysics-P2P) loss in the earth mover distance (EMD) space, applied iteratively. Experimental findings affirm that our technique surpasses multiple benchmarks in dough manipulation, spanning both short and long horizons. Remarkably, our model demonstrates robust generalization capabilities to novel and previously unencountered complex tasks without any preliminary demonstrations. We further substantiate our approach with experimental trials on real-world robotic platforms. Our project page: https://qq456cvb.github.io/projects/donut.

Figures (12)

• Figure 1: Schematic illustration of our method in handling unseen dough making tasks, where Language Models (LLMs) are utilized at a high level for task decomposition and subgoal generation, specifying tool names and generating corresponding Python code. The low-level operates on particle space controls, precisely determining the next achievable candidate iteratively without the need for prior demonstrations or task-specific training.
• Figure 2: EMD-space planning with DiffPhysics-P2P. We find the next reachable target by running small steps within the EMD space. The induced point-to-point correspondence can provide better gradients when optimizing actions.
• Figure 3: Illustration on how tool reset works. By resetting the tool position when no improvement can be made, we can jump out of the local minima.
• Figure 4: "Make a Donut." An exemplary zero-shot execution on complex long-horizon tasks.
• Figure 5: Cut, spread, arrange. Single-tool execution results.